Homotopy-Theoretic and Categorical Models of Neural Information Networks — Basic Neuroscience and Geometry of Neuroscience

This lecture is based on Yuri Manin and Matilde Marcolli, Homotopy-theoretic and categorical models of neural information networks, Compositionality 6 (2024), no. 4, 1–86. Related work: M. Marcolli, Gamma spaces and information, Journal of Geometry and Physics 140 (2019), 26–55; M. Marcolli, Categorical Hopfield networks, arXiv:2201.02756; M. Marcolli, Pareto optimization in categories, arXiv:2204.11931.

Motivation

Nontrivial homology

Kathryn Hess’s applied topology group at EPFL carried out a topological analysis of neocortical microcircuitry for the Blue Brain Project. The slide shows their plot of the Euler characteristic (EC, on a scale of ) of the active part of the network against time (ms) after a stimulus, for several stimuli (labelled S5a–c, S15a–c, S30a–c): the Euler characteristic is near zero, climbs steeply to a sharp peak (above for the strongest responses) around – ms, and then falls back to near zero; the weaker responses peak lower and spread out later in time.

In response to stimuli, a large number of high-dimensional cliques of neurons form (complete graphs on vertices with a directed structure).
The formation of these structures is responsible for an increasing amount of nontrivial Betti numbers and Euler characteristics, which reaches a peak of topological complexity and then fades.
Proposed functional interpretation: this peak of nontrivial homology is necessary for the processing of stimuli in the brain cortex... but why?

Computational role of nontrivial homology

There is a mathematical theory of concurrent and distributed computing (Fajstrup, Gaucher, Goubault, Herlihy, Rajsbaum, ...). In it, the initial and final states of processes are vertices, and mutually compatible initial/final process states span a -simplex. (The slide shows such a simplicial complex: a subdivided polyhedral complex drawn inside a cube, with a few of its triangles shaded.)

Distributed algorithms are described by simplicial sets and simplicial maps.
Certain distributed algorithms require “enough nontrivial homology” to successfully complete their tasks (Herlihy–Rajsbaum).
This suggests a possible functional role of nontrivial homology: to carry out some concurrent/distributed computation?

Neural codes and homotopy types

Carina Curto and collaborators: the geometry of the stimulus space can be reconstructed up to homotopy from the binary structure of the neural code.

In the figure, a stimulus point lying in the place fields of neurons 3 and 5, and in no other, makes exactly those two neurons fire, giving the codeword . (A second picture on the slide shows a region covered by many overlapping place fields, drawn as dashed circles, together with the graph of their overlaps.)

The overlaps between place fields of neurons determine the simplicial complex of the open covering (its nerve), and this complex has the same homotopy type as the stimulus space.
This suggests that the neural code represents the stimulus space through homotopy types, so homotopy theory is a natural mathematical setting.

Informational and metabolic constraints

Neural codes: rate codes (the firing rate of a neuron), spike timing codes (the timing of spikes), neural coding capacity for a given firing rate, output entropy.
The metabolic efficiency of a transmission channel is the ratio of the mutual information of output and input to the energy cost per unit of time.
Optimization of information transmission is done in terms of the connection weights, maximizing the mutual information .
Requirement for homotopy-theoretic modelling: one needs to incorporate constraints on resources and information. (Mathematical theory of resources: Tobias Fritz and collaborators give a categorical setting for a theory of resources and constraints.)

Informational complexity

Measures of informational complexity of a neural system have been proposed, such as integrated information: over all splittings of a system, compute the mutual information across the two subsystems, and take the minimum over all such splittings. (The slide’s picture shows four model neurons with arrows for their connections and a dashed line cutting the network into two halves; the same network reappears in the section on the function below.)

Integrated information is the least amount of effective information in the whole system that is not accounted for by the effective information of its parts.
There is a controversial proposal (Tononi) of integrated information as a quantitative measure of consciousness.
Question: is there a mathematical description of the organization of a neural system over its subsystems from which integrated information follows?

Main idea

The main idea for a homotopy-theoretic modelling of neural information networks:

We want a (topological) space that describes all consistent ways of assigning to a population of neurons with a network of synaptic connections a concurrent/distributed computational architecture (“consistent” means with respect to all possible subsystems).
We want this space also to keep track of constraints on resources and information, of conversion of resources, and of transmission of information (and information loss) across all subsystems.
We want this description also to keep track of homotopy types (to have homotopy invariants, associated homotopy groups): topological robustness.
Why use category theory as the mathematical language? Because it is especially suitable to express “consistency over subsystems” and “constraints over resources”.
Categorical language is also a main tool in homotopy theory (and the mathematical theory of concurrent/distributed computing already knows this!).

The language of categories

Categories, functors, natural transformations

Definition G8.1 (Category). A category has two parts to its structure: objects , and morphisms between objects, forming sets . Morphisms compose: and give , and composition is associative:

In the diagrammatic view (string diagrams), objects are wires (possible states) and morphisms are processes; composing two processes on a wire gives a single process.

Definition G8.2 (Functor). A functor compares different categories: it is a map of objects, for , together with a compatible map of morphisms, for .

Key idea: morphisms describe how objects can be transformed, and functoriality means that objects are mapped compatibly with all their possible transformations.

Definition G8.3 (Natural transformation). Functors are compared by natural transformations. Two functors can be related by a natural transformation , which means:

for every object there is a morphism in ;
for every these are compatible, that is, the diagram commutes.

Sums, zero objects, monoidal structure

A categorical sum is an additional property of a category , which describes being able to form composite states/systems.

Definition G8.4 (Categorical sum). For all objects of there is a sum object . It is identified (up to unique isomorphism) by a universal property: there is a unique way of extending morphisms from and from to morphisms from . Namely, there are morphisms and such that, given any morphisms and , there exists a unique morphism that completes the commutative diagram

that is, and .

So if you know how to transform two subsystems , , this determines how to transform their composite system .

Definition G8.5 (Zero object). A zero object is an object that has a unique morphism to any other object and a unique morphism from any other object. The zero object (when it exists) represents a trivial system. In particular .

Definition G8.6 (Unital symmetric monoidal category). In a unital symmetric monoidal category the operation of forming composite systems is written as , with a unit object . Its diagrammatic rules are as follows (wires are objects and boxes are morphisms):

Categories of codes, probabilities and resources

Finite pointed sets

Example G8.7 (Finite pointed sets). The prototype example of a category with sum and zero object:

objects are finite sets with a choice of a base point, ;
morphisms are maps of finite sets that map base point to base point, with ;
the categorical sum is the union of the sets with the base points identified;
the zero object is a single point.

Remark. Pointed sets are an unavoidable nuisance of homotopy theory! (Just think of the base point as being there for computational purposes, while “meaning” is attached only to the rest of the set.)

Binary codes and neural codes

Example G8.8 (Category of binary codes). Let be an binary code of length with . The category of pointed codes of length has:

objects: codes that contain the -word ; we exclude the code consisting only of the constant words and (for reasons of nontrivial information);
morphisms: functions mapping the -word to itself (we don’t require them to come from maps of the ambient );
sum as for pointed sets, (glued along the zero word);
zero object: the code consisting only of the zero word.

The role of the zero word is like a reference point (for a neural code, the baseline when no activity is detected).

Remark. In coding theory one often uses another form of categorical sum (decomposable codes), but it changes the code length :

Definition G8.9 (Neural codes). Let be a time interval of observation, subdivided into basic units of time .

The code length is the number of basic time intervals considered.
The number of nontrivial code words is the number of neurons observed.
Each code word is the firing pattern of one neuron: a digit for each time interval that contained a spike, and otherwise.
The zero word is the baseline of no activity (for comparison).

A neural code for neurons is a sum with , where is the zero word and is the firing pattern of the -th neuron.

Finite probabilities

Example G8.10 (Finite probabilities and fiberwise measures). The category of finite probabilities with fiberwise measures has:

objects : finite pointed sets with a probability measure (with );
morphisms : pairs of a pointed function with and weights for , such that

Example G8.11 (Probability measure associated to a code). Let be a binary code of length , and let be the number of digits equal to in the word (the Hamming distance to the reference zero word ). Define the probability distribution

The mapping determines a functor .
On morphisms, take This is fine because the only code word with is the -word , and only for the word (assuming codes do not contain only and ).
The probability keeps track of the information transmitted by the code .

Weighted codes

Example G8.12 (Category of weighted codes). The category of weighted binary codes has:

objects: pairs of a code of length containing the zero word and a function assigning a (signed) weight to each code word, with ;
morphisms : a map mapping the zero word to itself with , and weights for ;
sum with and ;
zero object .

Categories of resources

The mathematical theory of resources is B. Coecke, T. Fritz, R.W. Spekkens, A mathematical theory of resources, Information and Computation 250 (2016), 59–86 [arXiv:1409.5531].

Definition G8.13 (Category of resources). Resources are modelled by a symmetric monoidal category :

objects represent resources; the product represents the combination of resources; the unit object is the empty resource;
morphisms in represent possible conversions of resource into resource ;
is convertible into when .

The following notions are from Coecke, Fritz, Spekkens.

Definition G8.14 (Preordered semigroup, conversion rate, measuring semigroup).

The category determines a preordered abelian semigroup on the set of isomorphism classes of : is the class of , the unit is given by the unit object , and if and only if . (The same works for a category with sum and zero object.)
The maximal conversion rate of resources is the maximal number of copies of that can be produced, on average, from each copy of .
A measuring semigroup is an abelian semigroup with a partial ordering, together with a semigroup homomorphism with in whenever in .

A measuring semigroup satisfies (for real-valued : gives ).

Constraints and optimization in categorical form

Constraints and optimization can be expressed through limits and colimits in categories. Given a diagram and a cone over it, the limit is the “optimal cone”: every cone factors uniquely through it (dually for colimits).

Special cases of limits and colimits are equalizers and coequalizers.

Example G8.15 (Thin categories). A thin category has set of objects and one morphism whenever .

A diagram in is a selection of a subset .
Limits and colimits are greatest lower bounds and least upper bounds of subsets .

Key idea: functors compatible with limits and colimits describe constrained optimization.

Example G8.16 (Optimization of resources: adjoint functors). Let be a functor that assigns resources to (computational) systems. Suppose there is a left adjoint , so that The adjoint functor is a solution to an optimization problem: the assignment is an optimal way of associating a computational system in the category to given constraints on available resources, encoded by fixing .

The system is constructed from the resources : some of the resources are used for the manufacturing of , so one expects a conversion from to the remaining resources available to the system , namely . This is why we use a left rather than a right adjoint.

Theorem G8.17 (Freyd’s adjoint functor theorem). The condition for the existence of the left adjoint is the existence of a solution set: a family of systems (the systems optimal for resources ), with morphisms , such that for any system for which there is a possible conversion of resources in , there is one of the systems and a modification of systems in such that the conversion of resources factors through the system , namely

So the solution set consists of optimal systems from which any other system that uses fewer resources than can be obtained via allowed modifications (morphisms).

Summing functors and Gamma spaces

Summing functors

Summing functors come from Segal’s Gamma-spaces formalism (1973).

Definition G8.18 (Summing functor). Let be a category with sum and zero object (binary codes, transition systems, resources, etc.). Let be a pointed finite set and the category whose objects are the pointed subsets and whose morphisms are the inclusions . A functor is a summing functor if and is the zero object of .

is the category of summing functors , whose morphisms are the invertible natural transformations.

Key idea: a summing functor is a consistent assignment of resources of type to all subsystems of , so that a combination of independent subsystems corresponds to combined resources.
is the configuration space that parameterizes all possible such assignments.

The original formulation of Segal is for with sum and zero object; it was extended by Thomason to unital symmetric monoidal. Write for the base point of .

Proposition G8.19. A summing functor is completely determined by its values on the subsets , for . Let be the category with the same objects as and with morphisms the invertible morphisms of . If is a finite pointed set with , the category of summing functors is equivalent to .

Similarly, in the unital symmetric monoidal case summing functors are specified by in , with morphisms in .

Segal’s Gamma spaces

Gamma spaces are a construction introduced in homotopy theory in the ‘70s: a general construction of (connective) spectra (generalized homology theories).

Definition G8.20 (Gamma space). A Gamma space is a functor from finite (pointed) sets to (pointed) simplicial sets. (Here is the category of finite pointed sets and stands for the category of pointed simplicial sets.)

(The slide’s picture: a scattered finite set of points, and an arrow to a simplicial complex built on such points, with several coloured triangles and tetrahedra.)

Proposition G8.21. A category with sum and zero object determines a Gamma space : to a finite set assign the category of summing functors , and then the simplicial set given by the nerve of this category.

Meaning of this construction in our setting.

Definition G8.22 (Nerve of a category). In the nerve of a category, the vertices are the objects of the category, the edges are the morphisms between objects, and any chain of compositions of morphisms gives a simplex. For example, a 2-simplex:

The faces and degeneracies are given by composing successive morphisms in a chain and by inserting identity morphisms.

The nerve of the category of summing functors organizes all assignments of -resources to -subsystems and their transformations into a single topological structure that keeps track of the relations between them (morphisms of and their compositions become simplexes of the nerve).
So we view as a topological parameterizing space for all such consistent assignments of resources of type to subsets of .

Meaning in homotopy theory (loop–deloop).

Take the case where is an abelian category. Then (Quillen) the higher K-theory is an infinite loop space.
The category of summing functors provides a delooping of this infinite loop space (Carlsson).
A Gamma space defines an associated spectrum, by extending the functor to an endofunctor and applying it to spheres.
When , the Gamma space gives the sphere spectrum.
All connective spectra are obtained through this construction for a symmetric monoidal category (Thomason).
Hence the nerves are topologically very nontrivial.

From finite sets to networks

Directed graphs

Definition G8.23 (Directed graphs as functors). Let be the category with two objects and two (non-identity) morphisms , and let be the category of finite sets (objects finite sets, morphisms functions between finite sets). A directed graph is a functor :

is the set of edges of the directed graph;
is the set of vertices of the directed graph;
and are the usual source and target maps of the directed graph.

The category of directed graphs is : its objects are these functors and its morphisms are natural transformations.

Network summing functors

We want to replace the finite set with a directed graph (a functor from the category with objects and morphisms to finite sets). The basic additivity rule is for a functor from subgraphs with inclusions: Then we want additional compositionality rules:

conservation laws at vertices (Kirchhoff): equalizer and coequalizer constructions for source and target;
grafting operation (when the target category has an operad or properad structure);
inclusion–exclusion (as exact sequences) when is abelian;
etc.

Example G8.24 (Conservation laws at vertices). There are source and target functors sending a summing functor on subsets of edges to the summing functors and on subsets of vertices. The equalizer is a category with a functor such that , with the universal property that any with factors as :

This is the category of summing functors with a conservation law at vertices: for all , In particular, for all the inflow of -resources equals the outflow: Another kind of conservation law is expressed by a coequalizer.

Example G8.25 (Grafting). Let be a symmetric monoidal category with full subcategories such that

;
the monoidal structure satisfies
the family is a properad in .

Then consists of the summing functors with:

;
, where is the corolla of ;
, where is the set of edges with one end in and one in , and is the properad composition

Here and count the incoming and outgoing external edges of the subgraph , the corolla is the vertex with its incident edges, and is the subgraph obtained by joining and along the edges .

Proposition G8.26. A network summing functor is completely determined by its values on corollas.

Gamma spaces for networks

There is a functor with , the nerve of the equalizer of (an equalizer of nerves).
There are more general types of Gamma networks besides the equalizers (and coequalizers):
- for , take the category with objects the (pointed) subgraphs of and morphisms the (pointed) inclusions ;
- take the category of summing functors ;
- now the value of a functor on a subnetwork is not just the sum of its values on the edges in the subnetwork;
- there is a possibly more complicated dependence on the network structure (beyond conservation at vertices): general inclusion–exclusion type properties.
We focus on the equalizer case for simplicity.

Example G8.27 (Equalizer: linear model (resources: weighted codes)). A summing functor in the equalizer of the source and target functors, is a summing functor with the conservation laws for . Write . If the directed graph has a single outgoing edge at each vertex, , then the equalizer condition can be seen as a kind of categorical version of a linear neuron model.

We need nonlinearities: thresholds and saturation.
We need a formulation analogous to chamber structures (hyperplane arrangements) and different regions of linearity in the categorical setting.

Things you can do with this

Gamma spaces have associated topological invariants (homotopy groups), so one can decide whether assigning resources of type or of type to a network is “inequivalent” on the basis of these invariants: for some .
Any (functorial) transformation of resources carries over the whole structure.
describes resource assignments of binary codes to a network (neural codes). Passing from codes to probabilities gives information constraints on the network; functorially assigning computational architectures (transition systems) to codes gives the computational resources of the network; assigning resources to computational systems gives metabolic constraints on the computational power of the network, etc.
All these levels of structure and their relations should transform consistently in the dynamics.

Dynamics: categorical Hopfield networks

From kinematics to dynamics

So far this is just an “empty stage”: as in the modelling of physical systems, kinematics sets the stage but dynamics gives the actual model.
Gamma spaces are just kinematics for consistent assignments of resources to networks.
We need to make this setting dynamical to obtain actual models.
Key idea: extend the Hopfield equation in networks to a category-theoretic version for resources assigned to networks.
Fix the network geometry and take time-variable resources (codes, weights, probabilities, information structures, computational and metabolic resources, etc.), all evolving consistently with their functorial relations.
The resulting category-theoretic dynamics also induces a topological dynamical system on the nerve space .
Further steps? Making the network itself dynamical...

Discrete and continuous Hopfield dynamics

Definition G8.28 (Hopfield dynamics). Discrete version (binary neurons):

Continuous version (neuron firing rates as variables, threshold-linear dynamics): where are real-valued connection strengths, are constant external inputs, and is the threshold function.

Finite difference version: There are versions with or without the “leak term” on the right-hand side.

Definition G8.29 (Hopfield network). There is a historical connection between the statistical physics of spin glass models and neural networks. In a Hopfield network the node variables are , with update rule and energy

The dynamics converges to a fixed point attractor: this is the Hopfield network as a model of associative memory.

Hopfield networks with threshold nonlinearity

Recent work by Carina Curto and collaborators:

The Hopfield equations (continuous version) with threshold nonlinearity are a patchwork of linear systems across wall crossings given by a hyperplane arrangement.
Fixed points arise when fixed points of the linear system lie in the correct chamber of the hyperplane arrangement.
The dynamical behavior (fixed point supports) comes from combinatorial structures of (sub)graphs, “motifs”.

(The slide shows a hyperplane arrangement in three dimensions: several coloured planes cutting through a cube.)

Categorical Hopfield dynamics

Here we follow a different (but related) idea: Hopfield equations where the variables are not the real numbers but assignments of resources to the network (summing functors ), that is, an equation on the configuration space . Key idea: all levels of structure associated to the network (functorially related) evolve consistently in the dynamics.

The dynamics takes place in a category of network summing functors.
Possibly there is a functor to another category of resources, with respect to which the threshold of the dynamics is measured (one can take for simplicity).
There are two main ingredients for a categorical form of the Hopfield equations: the weights matrix is replaced by a system of endofunctors, and the threshold is replaced by a threshold endofunctor.

Step 1.

As above, take for a network and a category of resources .
Let be a functor to another category of resources (maybe the same) with respect to which the dynamics is measured, and let be the preordered semigroup of the category .
For threshold dynamics we use the relation , where is the class of .
Let be the category of monoidal endofunctors of , with natural transformations as morphisms. The sum of endofunctors is defined pointwise: for all .

Definition G8.30 (Threshold endofunctor). The threshold endofunctor is It is not an endofunctor of , and is not monoidal.

A summing functor, given in terms of , yields a new summing functor with values in for .

Step 2.

Definition G8.31 (Bisumming functors).

Bisumming functors are functors that are summing in both arguments.
In coordinates: , with .
is the category of bisumming functors with invertible natural transformations.
is the equalizer of the functors

Step 3.

Definition G8.32 (Categorical Hopfield equation). The equation is or the variant (with the leak term or not)

Initial condition: ; set (or just ).
A fixed summing functor ; set .
Take , and let be the class in of the object in .
The threshold gives if , and the zero object of otherwise.

For the case of it is better to use the equation at vertices (where the operad/properad composition happens): or

Proposition G8.33 (Some properties of the dynamics).

defines a sequence of summing functors in .
The assignment defined by the solution defines an endofunctor .
This induces a discrete topological dynamical system on the realization (Segal’s Gamma space).
For with a measuring semigroup, the categorical Hopfield dynamics induces the usual (finite difference) Hopfield dynamics on the weights.

Question: are there general results in the categorical setting about existence of solutions and behavior?

Example: neurons modelled by deep neural networks

This example models the neurons (nodes of the network) with DNNs, following D. Beniaguev, I. Segev, M. London, Single cortical neurons as deep artificial neural networks, Neuron 109 (2021) 2727–2739. The slide reproduces their figure: an L5 cortical pyramidal neuron (with apical, oblique and basal dendrites marked) shown as equivalent to a deep network with an input layer, hidden layers and an output, together with voltage traces comparing the neuron with the model; one panel uses a DNN with seven hidden layers and the other a network with one hidden layer of 128 nodes.

The category of DNNs

Definition G8.34 (Interfaced directed acyclic graphs). Interfaced directed acyclic graphs (IDAGs) are finite acyclic graphs with vertex set (inputs, hidden nodes, outputs) and source and target maps and .

Category : objects are IDAGs , and morphisms are morphisms of directed graphs (mapping inputs to inputs and outputs to outputs).
Properad structure: is the full subcategory with inputs and outputs; the composition rules graft the set of outputs to the set of inputs .
Operad: the single-output case (the case relevant for the model), .
Monoidal structure on : grafting at the output. For and ,

This is the underlying graph architecture of the DNNs; then we need weights.

Definition G8.35 (Weighted single-output IDAGs). The category of weighted single-output IDAGs has:

objects: pairs with an IDAG with a single output and (the weight matrix) a function , with the vertices, satisfying if (there are no looping edges, so for all );
morphisms : morphisms of directed graphs with , the pushforward of the weight matrix,
monoidal structure with
unit ;
operad structure as in , with composition on weights: when ; when ; and if the output is grafted to the input , the weights are and ; zero otherwise.

Proposition G8.36 (Inhibitory–excitatory balance). The unit is neither initial nor final in :

the existence of a morphism implies that is the trivial map (zero weights);
the existence of a morphism implies an inhibitory–excitatory balance condition over the whole network .

Proof. For the second statement: the map collapsing the graph to the single output vertex satisfies , and is the sum of all the weights . ∎

Dynamics with and without leak term

Proposition G8.37 (Dynamics with leak term). Consider an endofunctor of the category that maps with a summing functor defined by local data and an endofunctor with local data , and write for the term added at the -th step. Regardless of the endofunctors , solutions converge to a fixed point (in finite time) if and only if the inhibitory–excitatory balance on is violated at some step; otherwise and there is no limit.

Proof. Stationary solutions are fixed points of the endofunctor : objects with an isomorphism Already on the underlying directed graphs there is an isomorphism where is the graph of . By the form of in (grafting at the output, so that ), this forces , that is, . Taking to be the identity, the threshold condition means that there is a morphism , which by the previous proposition is the inhibitory–excitatory balance condition. So a step leaves unchanged exactly when the balance on is violated; at every other step the number of vertices grows. ∎

Dynamics case without leak term (more interesting). The evolution equation is

Stationary solutions are pairs where the right-hand side satisfies the balance condition and
If the balance condition is violated at some step, the solution stabilizes at .
Now the behavior depends on the specific choice of endofunctors .
Example: with . Think of the functor as implementing a backpropagation mechanism on the DNN machine .

Example G8.38 (Updating weights by gradient descent). Take for some cost function for the specific DNN and a scale (the learning rate).

Then a fixed point is a critical point of the cost function (a minimum, depending on the shape of ).
The consistency condition (for ) holds, since gives

Note: this is an oversimplistic toy model, as it assumes the updating mechanism of the weights of each DNN machine is uncoupled from the other machines at other nodes.
These are just consistency checks: we get the usual Hopfield equations, and we get DNN backpropagation.
Other related cases: more general automata (transition systems, models of concurrent and distributed computing).

Example: concurrent and distributed computing architectures

Transition systems

The category of transition systems is from G. Winskel, M. Nielsen, Categories in concurrency, in “Semantics and logics of computation (Cambridge, 1995)”, pp. 299–354, Publ. Newton Inst. 14, Cambridge Univ. Press, 1997. These are models of computation that involve parallel and distributed processing.

Definition G8.39 (Category of transition systems).

Objects are , with the set of possible states of the system, the initial state, a set of labels, and the set of possible transitions, (each specified by initial state, label of the transition, final state). Equivalently, is a directed graph with vertex set and with set of labelled directed edges .
Morphisms of transition systems are pairs of a function with and a (partially defined) function of labelling sets, such that for any transition in , if is defined, then is a transition in .
The categorical sum is where both sets are seen as subsets of
The zero object is the stationary single-state system with empty labels and transitions.

Definition G8.40 (Grafting of transition systems). Let , for , be objects of the category of transition systems, and choose two states and . The grafting in has , , and .

Let be the subcategory of transition systems with a single final state . Then the grafting is given by , with the final state of grafted to the initial state of .

Proposition G8.41. If is a finite acyclic directed graph and a topological ordering of its vertex set , then given objects of there is a well-defined grafting of the , which is also an object of .

Strongly connected components and the computational architecture functor

Definition G8.42 (Strongly connected components, condensation graph). Let be a finite directed graph. A subset is a strongly connected component if each vertex in is reachable through an oriented path in from any other. The condensation graph is the directed acyclic graph obtained from by contracting each subgraph of a strongly connected component to a single vertex.

Definition G8.43 (Computational architecture functor). Let be the category of finite directed graphs.

is the category with objects the pairs with and , and morphisms with and .
is the subcategory of with objects where the summing functor takes values in .
The functor assigns to an object the grafting , along the condensation graph , of the , with the strongly connected components of .

Modelling computational architectures of neuronal networks

Local automata model (discretized) individual neurons with pre-synaptic and post-synaptic activity.
The grafting of these automata, where their inputs and outputs are connected, models the connectivity of the network.
One can adapt this setting to model non-local neuromodulation effects (distributed computing models of neuromodulation: Potjans–Morrison–Diesmann).

Keeping track of measures of informational complexity

Information and codes

There is a probability distribution associated to neural codes through the firing rate.
A word of length records a digit for each time interval that contains a spike and a otherwise.
Consider with and the Bernoulli measure: on the set of sequences beginning with the word , with the number of ’s and the number of zeros.
For large a neural code is in the Shannon Random Code Ensemble of :

Recall the category of finite probabilities with fiberwise measures (not normalized) as morphisms: with a pointed map with , and measures on the fibers , with and

has a zero object and sum.
There is a functor ,
So consistent assignments of codes to a network give an assignment of probabilities.
Information: let be the subcategory where are surjections and the , for , are probability measures. Then the Shannon entropy is a functor (with viewed as a thin category).

Some questions

Can more interesting examples of categorical Hopfield dynamics be constructed?
What about other categories of resources?
What about other models of categorical dynamics (related to Hopfield network dynamics)?
About this last question: searching for equilibria of Hopfield networks is related to searching for Pareto optimization solutions.
Next question: is there a categorical Pareto optimization?

Integrated information

References for integrated information (Tononi):

G. Tononi, Consciousness as integrated information: a provisional manifesto, Biol. Bull. 215 (2008) no. 3, 216–242.
M. Oizumi, N. Tsuchiya, S. Amari, Unified framework for information integration based on information geometry, PNAS 113 (2016) no. 51, 14817–14822.

We want to measure the amount of informational complexity in a system that is not separately reducible to its individual parts.
It concerns the possibilities of causal relatedness among different parts of the system.

Computing integrated information.

Consider all possible ways of splitting a given system into subsystems.
The state of the system at a given time is described by a set of observables , and the state at a near-future time by .
A partition into subsystems gives a splitting of these variables, and , into variables describing the subsystems.
All causal relations among the or among the , and also the causal influence of the on the through time evolution, are captured (statistically) by the joint probability distribution .
Compare the information content of this joint distribution with a distribution where there are only causal dependencies between and through evolution within separate subsystems, not across subsystems.

Definition G8.44 (Integrated information (information-geometric form)). Let be the set of probability distributions with the property that for each subset of the partition . Minimize the Kullback–Leibler divergence between the actual system and its best approximation in , over the choice of partition : the integrated information is

The value represents the additional information in the whole system that is not reducible to smaller parts.

Cohomological view of information

This is due to Bennequin, Baudot and Vigneaux.

There is an abelian category describing probability data: the category of finite information structures, with random variables and a simplicial set of associated probabilities, with a functor to vector spaces (real-valued measurable functions); the result is an abelian category of modules over a sheaf of algebras.
It has a Hochschild cochain complex and associated cohomology.
Shannon entropy, KL divergence and Tsallis entropy all have an interpretation as nontrivial -cocycles, generators of the first cohomology.

Use this setting to construct:

a contravariant functor ;
using the construction above, a functor from to cochain complexes and cohomology;
using the Hochschild cocycle interpretation of the KL divergence, a cohomological interpretation of integrated information, with a functorial map from .

Further steps and proposed view

Further steps.

Neural codes generate homotopy types, in the form of the nerve simplicial set of an open covering associated to a (convex) code.
Recover that homotopy type from the above setting with information structures.
Combine the simplicial sets obtained in this way with those obtained via Gamma spaces describing assignments of resources to the network.
The simplicial sets given by the clique complex of a network are also realized as a special case of finite information structures.

Conclusion: proposed view.

Working hypothesis: the brain represents the stimulus space through a homotopy type.
Mathematical modelling of network architectures in the brain should include mechanisms that generate homotopy types (Gamma spaces, information structures).
Higher topological complexity in these homotopy types implies (but is not implied by) higher values of (cohomological) integrated information.
Question: is there a good model of “qualia” in terms of homotopy types?

Integrated information and the problem(s) of consciousness

David Chalmers’ Hard Problem of Consciousness: explain why consciousness exists (how neuronal activity gives rise to “qualia”).
Qualia: individual instances of subjective conscious experience, “what it is like” to have a certain experience, “the ways things seem to us” (Dennett).
Consciousness tests: identifying and quantifying consciousness. To which physical systems can we attribute (a degree of) consciousness? This has serious ethical implications (for human, animal and machine).
The so-called easier problems of consciousness (easier = functionally definable): how the brain processes and integrates information, categorizes and discriminates environmental stimuli, or focuses attention.

The integrated information proposal (Tononi’s function)

References:

G. Tononi, An information integration theory of consciousness, BMC Neurosci. 5 (2004) 42.
G. Tononi, Consciousness as integrated information: a provisional manifesto, Biol. Bull. 215 (2008) 216–242.
G. Tononi, C. Koch, The neural correlates of consciousness: an update, Ann. N.Y. Acad. Sci. 1124 (2008) 239–261.
G. Tononi, O. Sporns, G.M. Edelman, Measures of degeneracy and redundancy in biological networks, Proc. Natl. Acad. Sci. USA 96 (1999) 3257–3262.
D. Balduzzi, G. Tononi, Qualia: the geometry of integrated information, PLoS Comput. Biol. 5 (2009), no. 8, e1000462, 24 pp.

Some general requirements for a measure of consciousness. (The slide illustrates each requirement with a picture of a visual scene inside a circle: the same scene for intrinsic existence; the scene with parts labelled, such as “blue”, “book”, “left”, for composition; the scene as one of a stack of alternative scenes for information; the scene cut in two by a dashed line for integration; and the scene with a sharp boundary inside a dashed outer region for exclusion.)

Is there a good set of “axioms” for consciousness? The proposal is:

Intrinsic existence: an intrinsic perspective exists, independent of external observers.
Composition: it is structured into many levels, from elementary to higher order.
Information: differentiation between a specific set of elements in a specific experience and other possible alternatives.
Integration: it is unified, with no entirely independent subsets.
Exclusion: experience has definite content and spatio-temporal location.

Can these capture the essential nature of conscious experience?
Can one approach the question computationally?

Main heuristic idea about the function.

Key idea: consider all subdivisions of a physical system into two parts and ; over all such splittings, minimize a measure of mutual information between outputs and inputs.
Claim: consciousness corresponds to high values of this function.
Appealing aspect: consciousness is not just a yes/no state, but varies by degrees.

Preliminaries: information and mutual information

Definition G8.45 (Shannon entropy). For a probability on a finite set , the Shannon entropy is also often denoted by .

Theorem G8.46 (Khinchin axioms). Axiomatic characterization (Khinchin axioms):

is continuous on the simplex ;
is invariant under permutations of the ;
is maximal at the uniform distribution ;
compatibility: ;
extensivity (additivity over subsystems): for , where is a probability distribution on points and stands for the entries .

These properties determine uniquely up to a positive multiplicative constant .

Why is a measure of information? The maximal entropy case (the uniform distribution) contains the least information, while a supported at a single point has (no uncertainty). Using and taking as frequencies: for fair coin tossing, the entropy in number of bits is the base logarithm of the number of possible outcomes. The slide illustrates this with two coins and their four equally likely outcomes, so entropy measures the amount of uncertainty.

Definition G8.47 (Relative entropy: Kullback–Leibler divergence). For two probability distributions , the expectation of the log difference between the probabilities , with the expectation taken with respect to .

Properties:

it is always non-negative, (zero if and only if );
it is biconvex in ;
it is not a distance function: it is non-symmetric and can take the value ;
the leading term of its expansion is a distance function, the Fisher metric:

Definition G8.48 (Mutual information). The mutual information of and is With the conditional probability , the conditional entropy is

Equivalently, the mutual information is where is the Shannon entropy.

Inclusion–exclusion property of mutual information.

where is the Shannon entropy.

The function: integrated information

Definition G8.49 (Integrated information).

Consider all splittings of a system (e.g. a network) into two subsets (e.g. two disjoint sets of nodes, and look at the outputs of that are inputs of and vice versa).
Assume a discrete system whose states are described by a string in an alphabet (e.g. a binary string).
Assume the states of the system evolve according to a discrete dynamical system that updates the state: .
A subdivision splits the input into the parts of the variables in and in , and .

The effective information is obtained by drawing the input variables uniformly at random, fixing the values, and computing the Shannon entropy of the resulting output variables ; similarly for in the other direction:

Symmetrized form: .
Normalization: this corrects for the case where one set is very large and the other very small, like and .
Integrated information (with or without normalization?):

A closer look at the effective information

The slide shows a figure from Balduzzi and Tononi’s work: the four-neuron network from the motivation, with current outputs , cut by a dashed line into the parts and (the connections crossing the cut are greyed out and replaced by noise, firing with probability ). Next to it are bar charts over the four states of each part: the potential repertoires , (uniform, height on each of the four states) and the actual repertoires , (concentrated on fewer states), with bits for the first part and bits for the second.

These are the Kullback–Leibler divergences and of the two parts of the partition, with and the maximal entropy uniform distributions: this gives the effective information generated in each of the two disconnected regions.

To compute the effective information, then compute of the distribution on the whole system and of the partition. (In the slide’s second figure, all connections of the network are drawn; the actual repertoire of the whole system, over its states, is concentrated on a single state with probability , while the product for the minimal partition is spread over several states, giving bits.)

Then the integrated information is the least amount of effective information in the whole system that is not accounted for by the effective information of its parts.

Computability issues

For a system of size there are possible splittings into non-empty complementary subsets, and for each of these one needs to compute the effective information.
In addition to the exponential size of the computation, the effective information requires some explicit knowledge of the function (not feasible if is the interconnection network of the human brain).
Possible application: take binary variables given by the firing of a population of neurons (a binary neural code). There are some issues about the correct way to randomize variables in this case (Shannon random code ensemble?).
Conjecture (Aaronson): approximating to some assigned precision, with a polynomial-size Boolean circuit, is computationally NP-hard.

Necessary or sufficient? The Vandermonde matrix (Aaronson’s example)

Let , for some large prime , be with a Vandermonde matrix:

The Vandermonde matrix has full rank (maximal rank), and the can be chosen so that every submatrix has full rank.
There are more general constructions of matrices over finite fields with no singular square submatrices: J. Lacan, J. Fimes, A construction of matrices with no singular square submatrices, in “Finite fields and applications”, pp. 145–147, Lecture Notes in Comput. Sci. 2948, Springer, 2004.
For example, using Cauchy matrices for suitable , :

Lemma G8.50. Let and be two matrices over of rank such that any submatrix of the matrix has rank . Then has all square submatrices invertible.

Example G8.51. Let be the Vandermonde matrix of distinct (with columns indexed by the , so ), and with , where is a vector of distinct nonzero elements of and for all . Then , are as in the lemma, and has all square submatrices invertible.

Proposition G8.52 (Aaronson). Systems with realized by one of these matrices have the property that every subsystem preserves all the possible information. Aaronson shows that in this case any bipartition gives hence the integrated information is a constant, the maximal possible value, for each partition . The non-normalized varies between and (the entire information content of in -ary bits).

Reed–Solomon codes

What is going on here? The Vandermonde matrix maps a vector in , viewed as the coefficients of a polynomial of degree , to the list of values of this polynomial at the points of : this is the case of a Reed–Solomon code.
It is well known that a very good coding property of Reed–Solomon codes is that every subset of outputs of the code is as strongly correlated to every subset of inputs as possible (which is why codes from algebraic curves are so good!).

Definition G8.53 (Algebro-geometric codes).

Take the algebraic points of a curve over a finite field .
Take a set and a divisor on with .
The code is obtained by evaluation at of rational functions with poles at .
The bound on the order of the poles of at determines the dimension of the linear code.

Example G8.54 (Reed–Solomon codes). This is the case .

gives an code with .
Or use homogeneous polynomials at points :

Problem: examples constructed via coding theory (algebro-geometric codes) show that it is possible to generate simple systems that have extremely high values of integrated information (corresponding to maximal correlation of all output subsystems to all input subsystems).

While one can argue this may be a necessary condition for consciousness, it is hard to argue it is sufficient (are Reed–Solomon codes conscious entities?).
Assuming high integrated information is a reasonable necessary condition for the existence of consciousness, what are the obstructions to it being sufficient?

Qualia and integrated information

The proposal is in D. Balduzzi, G. Tononi, Qualia: the geometry of integrated information, PLoS Comput. Biol. 5 (2009), no. 8, e1000462, 24 pp. Some key ideas:

qualia space (probabilities on the set of states);
an experience as a shape in qualia space;
the quantity of consciousness (measured by );
the same system in different states may generate different qualia.

The slide reproduces their figure. Panel A: a system of four nodes, with node 1 highlighted and arrows between the nodes; “qualia space (16-dim) parametrizes repertoires of the system”, drawn as a fan of axes labelled by the states . Going from the uniform repertoire over all states to the repertoire specified by the connection between nodes and , , is an arrow in qualia space carrying bits. Panel B: the quale as a shape in qualia space, a polytope-like web of points joined by arrows, each point being the repertoire produced by a set of connections (drawn as small network icons, with the repertoires as bar charts around it), and each arrow labelled by the information it adds (, , , , , , and bits), up to the repertoire of the whole system, a single spike.

Definition G8.55 (Qualia space).

The qualia space for a system of elements: take the set of states of , which is ; then take the simplex of all probability distributions on the set of states .
Start with the maximum entropy , assigning equal probability to all states (the “bottom of the quale”).
Assign certain connections (edges) between the states, the corresponding maps , and a current state of the system.
This reduces the ambiguity of the initial state, as only certain initial configurations can lead to through the assigned connections and maps (the “mechanism”).
The posterior is determined by the current state and the mechanism (how compatible previous system states are with the system mechanism and the current state).
The effective information is the relative entropy of with respect to : with the Kullback–Leibler divergence and the Shannon entropy.

Example G8.56. Take elements and states, with the mechanism: if then set . If the current state is , then we know that the initial state could only be or (with equal probability , as there is no distinguishing factor between these), while the initial states and must have probability (they are not compatible with the output state ). So the effective information is bit.

The lattice of submechanisms

The slide shows (from Balduzzi–Tononi) the following example:

(A): a system of nodes and connections;
(B): the connections grouped into four submechanisms ;
(C): the lattice of combinations of the four submechanisms;
(D): the union ;
(E): ;
(F): as the complement of : in the four-element sublattice , one has , and .

Some geometric configurations in the lattice of submechanisms:

(G): paths from the bottom to the top;
(H): all mechanisms for varying ;
(I): the down-set of and the dual up-set of the complement of .

When a submechanism is included in another, the difference between the product of independent probabilities and the actual of the union is measured by
The quale is the set of all , for all submechanisms , together with the mappings between them that correspond to inclusions , weighted by .
A concept is “an indivisible informational relationship”: a discrimination performed by some mechanism in context that cannot be decomposed into a product of simpler discriminations.
High interdependence (not decomposable) is a requirement for a mechanism to generate a concept.
This all sounds appealing, but the same Aaronson objection holds: mathematical objects exhibiting high interdependence (see the coding theory examples) do not have qualia experiences and consciousness (or do they?).

Is topology helpful?

Cohomological aspects of mutual information: P. Baudot, D. Bennequin, The homological nature of entropy, Entropy 17 (2015) no. 5, 3253–3318.

The inclusion–exclusion property of mutual information is not accidental: it is an instance of cohomological properties of information functions.
There is a more general structure for higher mutual information functions:

Hochschild homology and cohomology

Definition G8.57 (Hochschild homology). Let be an associative algebra and a bimodule over . Set the -fold tensor product of over the ground field is also a bimodule. Define and The Hochschild homology is .

Definition G8.58 (Hochschild cohomology). Let be an associative algebra and a bimodule over . Set with and, for , Then , and the Hochschild cohomology is .

Example G8.59. The -cocycles are the derivations, , and the -coboundaries are the inner derivations, ; so is the space of outer derivations.

Information cohomology

Definition G8.60 (Hochschild coboundary for information functions). The Hochschild coboundary for information functions is with averaged conditioning A variant is the coboundary , with the first term replaced by .

Theorem G8.61 (Baudot–Bennequin). The higher mutual informations are Hochschild cocycles: with the Shannon entropy.

Question: is there a cohomological formulation of effective information and the function?

Why useful? ...It may help in addressing the NP-hard question and computational complexity questions for the function, and in finding some reasonable approximation procedure (filtrations of homology, etc.).
For comparison, in the simplicial world there are known computational complexity results for cycles and homologies:
- whether a simplicial complex is collapsible is NP-complete (Martin Tancer, Recognition of collapsible complexes is NP-complete, arXiv:1211.6254);
- whether a simplicial complex is contractible is undecidable;
- but computing simplicial homology has efficient algorithms (J.G. Dumas, F. Heckenbach, D. Saunders, V. Welker, Computing simplicial homology based on efficient Smith normal form algorithms, Algebra, geometry, and software systems, 177–206, Springer, 2003).

Any other topological insight?

In the analysis of neural codes it has been shown that the homotopy type of the stimulus space can be detected from the (binary) neural codes, under a convexity hypothesis on the receptive fields.
If the neural code is responsible for creating an internal representation of a stimulus, then this representation may be in the form of the information about the homotopy type stored in the code.
This suggests a possible working hypothesis: the space of qualia is a space of homotopy types.
This hypothesis is independent of any relation to the function as a measure of consciousness.
Instead it suggests viewing consciousness as a mechanism associating qualia to external stimuli in the form of computing homotopy types.

References

Yu. Manin, M. Marcolli, Homotopy-theoretic and categorical models of neural information networks, Compositionality 6 (2024), no. 4, 1–86.
M. Marcolli, Gamma spaces and information, Journal of Geometry and Physics 140 (2019), 26–55.
M. Marcolli, Categorical Hopfield networks, arXiv:2201.02756.
M. Marcolli, Pareto optimization in categories, arXiv:2204.11931.
B. Coecke, T. Fritz, R.W. Spekkens, A mathematical theory of resources, Information and Computation 250 (2016), 59–86, arXiv:1409.5531.
G. Winskel, M. Nielsen, Categories in concurrency, in “Semantics and logics of computation (Cambridge, 1995)”, pp. 299–354, Publ. Newton Inst. 14, Cambridge Univ. Press, 1997.
D. Beniaguev, I. Segev, M. London, Single cortical neurons as deep artificial neural networks, Neuron 109 (2021), 2727–2739.
G. Tononi, An information integration theory of consciousness, BMC Neurosci. 5 (2004), 42.
G. Tononi, Consciousness as integrated information: a provisional manifesto, Biol. Bull. 215 (2008) no. 3, 216–242.
G. Tononi, C. Koch, The neural correlates of consciousness: an update, Ann. N.Y. Acad. Sci. 1124 (2008), 239–261.
G. Tononi, O. Sporns, G.M. Edelman, Measures of degeneracy and redundancy in biological networks, Proc. Natl. Acad. Sci. USA 96 (1999), 3257–3262.
D. Balduzzi, G. Tononi, Qualia: the geometry of integrated information, PLoS Comput. Biol. 5 (2009), no. 8, e1000462.
M. Oizumi, N. Tsuchiya, S. Amari, Unified framework for information integration based on information geometry, PNAS 113 (2016) no. 51, 14817–14822.
J. Lacan, J. Fimes, A construction of matrices with no singular square submatrices, in “Finite fields and applications”, pp. 145–147, Lecture Notes in Comput. Sci. 2948, Springer, 2004.
P. Baudot, D. Bennequin, The homological nature of entropy, Entropy 17 (2015) no. 5, 3253–3318.
M. Tancer, Recognition of collapsible complexes is NP-complete, arXiv:1211.6254.
J.G. Dumas, F. Heckenbach, D. Saunders, V. Welker, Computing simplicial homology based on efficient Smith normal form algorithms, in “Algebra, geometry, and software systems”, 177–206, Springer, 2003.
Also mentioned in the lecture: the Blue Brain Project topology work of Kathryn Hess’s group (EPFL); Herlihy–Rajsbaum on distributed computing and homology; Carina Curto and collaborators on neural codes and threshold-linear networks; Segal’s Gamma spaces (1973), with Thomason, Quillen and Carlsson; Potjans–Morrison–Diesmann on neuromodulation; Chalmers and Dennett on consciousness and qualia; Aaronson’s critique of integrated information.

G8 Homotopy-Theoretic and Categorical Models of Neural Information Networks