FORCE Layers

Base FORCE Layer

class tension.base.FORCELayer(*args, **kwargs)

Base class for FORCE layers

Parameters:

  • units (int) – Number of recurrent units.

  • output_size (int) – Dimension of the target.

  • activation (str or function) – Activation function applied to the state updates. Can be a string (i.e. ‘tanh’) or a function.

  • seed (int or None) – Seed for random initialization (i.e. weights and initial states). (Default: None)

  • g (float) – Gain parameter that controls the chaos (by default, only scales the strengths of the recurrent connections within the network at initialization). (Default: 1.5)

  • input_kernel_trainable (bool) – If True, sets the input kernel to be a trainable variable. (Default: False)

  • recurrent_kernel_trainable (bool) – If True, sets the recurrent kernel to be a trainable variable. (Default: False)

  • output_kernel_trainable (bool) – If True, sets the output kernel to be a trainable variable. (Default: True)

  • feedback_kernel_trainable (bool) – If True, sets the feedback kernel to be a trainable variable. (Default: False)

  • p_recurr (float) – Recurrent kernel initialization parameter where larger values means denser connections (more non-zero weights). Value must be in (0,1]. (Default: 1)

States:

  • a - 1 x self.units tensor containing the pre-activation neuron firing rates

  • h - 1 x self.units tensor containing the neuron firing rates

  • output - 1 x self.output_size tensor containing the predicted output

build(input_shape)

Inherited from tensorflow.keras.layers.Layer.build, also calls method that performs the initialization of layer kernals. Typically called implicitly.

Parameters:

input_shape (tuple) – Shape of the input tensor.

abstract call(inputs, states)

Implements forward pass of the layer.

Parameters:

  • inputs (Tensor[2D float]) – Input tensor of shape (1, input dimensions).

  • states (list[Tensor[2D float]]) – List of tensors corresponding to the states of the layer.

Returns:

  • output (Tensor[2D float]) - 1 x self.output_size tensor containing the output of the forward pass.

  • updated states (list[Tensor[2D float]]) - List of tensors containing the updated states of the layer.

classmethod from_weights(weights, recurrent_nontrainable_boolean_mask, **kwargs)

Creates a layer object with pre-initialized weights.

Note: p_recurr parameter is not supported in this method. units and output_size parameters are inferred from the input weights.

Parameters:

  • weights (tuple[Tensor[2D float]] of length 4) – Four 2D Tensors containing, respectively, the input, recurrent, feedback, and output kernels

  • recurrent_nontrainable_boolean_mask (Tensor[2D bool]) – A 2D boolean array with the same shape as the recurrent kernel, where True indicates that the corresponding weight in the recurrent kernel is not trainable.

  • kwargs – Additional parameters required to initialize the layer.

Returns:

A sub-classed FORCELayer object initialized with the input weights

abstract get_initial_state(inputs=None, batch_size=None, dtype=None)

Initializes the states of the layer (called implicitly during layer build). See: https://www.tensorflow.org/api_docs/python/tf/keras/layers/RNN

Returns:

A list of tensors containing the initial layer states

initialize_feedback_kernel(feedback_kernel=None)

Initializes the feedback kernel. Called in the layer object’s build and from_weights methods.

Parameters:

feedback_kernel (Tensor[2D float]) – Tensor array containing the pre-initialized kernel. If none, the kernel will be randomly initialized. (Default: None)

initialize_input_kernel(input_dim, input_kernel=None)

Initializes the input kernel. Called in the layer object’s build and from_weights methods.

Parameters:

  • input_dim (int) – Dimension of input

  • input_kernel (Tensor[2D float]) – Tensor containing the pre-initialized kernel. If none, the kernel will be randomly initialized. (Default: None)

initialize_output_kernel(output_kernel=None)

Initializes the output kernel. Called in the layer object’s build and from_weights methods.

Parameters:

output_kernel (Tensor[2D float]) – Tensor or numpy array containing the pre-initialized kernel. If none, the kernel will be randomly initialized. (Default: None)

initialize_recurrent_kernel(recurrent_kernel=None)

Initializes the recurrent kernel. Called in the layer object’s build and from_weights methods.

Parameters:

recurrent_kernel (Tensor[2D float]) – Tensor containing the pre-initialized kernel. If none, the kernel will be randomly initialized. (Default: None)

Echo State Networks

class tension.models.EchoStateNetwork(*args, **kwargs)

Implements the feedback echo state network. See base.FORCELayer for states.

Parameters:

  • dtdivtau (float) – dt divided by network dynamic time scale.

  • hscale (float) – A scaling factor for randomly initializing the initial network activities. (Default: 0.25)

  • initial_a (Tensor[2D float] or None) – An optional 1 x self.units tensor or numpy array specifying the initial neuron pre-activation firing rates. (Default: None)

  • kwargs – See FORCELayer for additional required args.

call(inputs, states)

Implements forward pass of the layer.

Parameters:

  • inputs (Tensor[2D float]) – Input tensor of shape (1, input dimensions).

  • states (list[Tensor[2D float]]) – List of tensors corresponding to the states of the layer.

Returns:

  • output (Tensor[2D float]) - 1 x self.output_size tensor containing the output of the forward pass.

  • updated states (list[Tensor[2D float]]) - List of tensors containing the updated states of the layer.

get_initial_state(inputs=None, batch_size=None, dtype=None)

Initializes the states of the layer (called implicitly during layer build). See: https://www.tensorflow.org/api_docs/python/tf/keras/layers/RNN

Returns:

  • a (Tensor[2D float]) - batch_size x self.units tensor containing the pre-activation neuron firing rates.

  • h (Tensor[2D float]) - batch_size x self.units tensor containing the neuron firing rates.

  • output (Tensor[2D float]) - batch_size x self.output_size tensor containing the predicted output.

class tension.models.NoFeedbackESN(*args, **kwargs)

Implements the no feedback echo state network. See base.FORCELayer for states.

Parameters:

  • recurrent_kernel_trainable (bool) – Boolean on whether or not to train recurrent kernel. (Default: True)

  • kwargs – See FORCELayer and EchoStateNetwork for additional required args.

build(input_shape)

Inherited from tensorflow.keras.layers.Layer.build, also calls method that performs the initialization of layer kernals. Typically called implicitly.

Parameters:

input_shape (tuple) – Shape of the input tensor.

call(inputs, states)

Implements forward pass of the layer.

Parameters:

  • inputs (Tensor[2D float]) – Input tensor of shape (1, input dimensions).

  • states (list[Tensor[2D float]]) – List of tensors corresponding to the states of the layer.

Returns:

  • output (Tensor[2D float]) - 1 x self.output_size tensor containing the output of the forward pass.

  • updated states (list[Tensor[2D float]]) - List of tensors containing the updated states of the layer.

classmethod from_weights(weights, recurrent_nontrainable_boolean_mask, **kwargs)

Creates a NoFeedbackESN object with pre-initialized weights.

Note: p_recurr parameter is not supported in this method. units and output_size parameters are inferred from the input weights.

Parameters:

  • weights (tuple[Tensor[2D float]] of length 3) – tuple of tensors containing the input, recurrent, and output kernels

  • recurrent_nontrainable_boolean_mask (Tensor[2D bool]) – A 2D boolean array with the same shape as the recurrent kernel, where True indicates that the corresponding weight in the recurrent kernel is not trainable.

  • kwargs – Additional parameters required to initialize the layer.

Returns:

A NoFeedbackESN object initialized with the input weights

class tension.constrained.ConstrainedNoFeedbackESN(*args, **kwargs)

Constrained ESN without feedback as described in Hadjiabadi et al..

Note: The target must have dimension equal to the number of neurons.

Parameters:

  • dtdivtau (float) – dt divided by network dynamic time scale.

  • structural_connectivity (Tensor[2D float]) – self.units x self.units structural connectivity matrix

  • noise_param (tuple[float]) – Tuple of length 2 containing (in order) the mean and standard deviation for the white noise in the forward pass.

  • initial_a (Tensor[2D float] or None) – An optional 1 x self.units tensor or numpy array specifying the initial neuron pre-activation firing rates. (Default: None)

  • recurrent_kernel_trainable (bool) – Boolean on whether or not to train recurrent kernel. Note that this is the only kernel that can be trained for this layer. (Default: True)

  • kwargs – See FORCELayer for additional required args. This layer has no output weights and therefore the output_size and output_kernel_trainable parameters are not supported.

build(input_shape)

Inherited from tensorflow.keras.layers.Layer.build, also calls method that performs the initialization of layer kernals. Typically called implicitly.

Parameters:

input_shape (tuple) – Shape of the input tensor.

call(inputs, states)

Implements forward pass of the layer.

Parameters:

  • inputs (Tensor[2D float]) – Input tensor of shape (1, input dimensions).

  • states (list[Tensor[2D float]]) – List of tensors corresponding to the states of the layer.

Returns:

  • output (Tensor[2D float]) - 1 x self.output_size tensor containing the output of the forward pass.

  • updated states (list[Tensor[2D float]]) - List of tensors containing the updated states of the layer.

classmethod from_weights(weights, recurrent_nontrainable_boolean_mask, **kwargs)

Creates a ConstrainedNoFeedbackESN object with pre-initialized weights.

Note: p_recurr parameter is not supported in this method. units parameter is inferred from the input weights.

Parameters:

  • weights (tuple[Tensor[2D float]] of length 2) – Two 2D Tensors containing the input and recurrent kernels

  • recurrent_nontrainable_boolean_mask (Tensor[2D bool]) – A 2D boolean array with the same shape as the recurrent kernel, where True indicates that the corresponding weight in the recurrent kernel is not trainable.

  • kwargs – Additional parameters required to initialize the layer.

Returns:

A ConstrainedNoFeedbackESN object initialized with the input weights

get_initial_state(inputs=None, batch_size=None, dtype=None)

Initializes the states of the layer (called implicitly during layer build). See: https://www.tensorflow.org/api_docs/python/tf/keras/layers/RNN

Returns:

  • a (Tensor[2D float]) - 1 x self.units tensor containing the pre-activation neuron firing rates

  • h (Tensor[2D float]) - 1 x self.units tensor containing the neuron firing rates

  • output (Tensor[2D float]) - 1 x self.units tensor containing the output of each neuron

initialize_recurrent_kernel(recurrent_kernel=None)

Initializes the recurrent kernel. Called in the layer object’s build and from_weights methods.

Parameters:

recurrent_kernel (Tensor[2D float]) – Tensor containing the pre-initialized kernel. If none, the kernel will be randomly initialized. (Default: None)

Spiking Networks

class tension.spiking.SpikingNN(*args, **kwargs)

Parent class part of the spiking neural networks as described in Nicola and Clopath. New spiking RNN layers can be created by subclassing from this class or spiking.OptimizedSpikingNN.

Note: The recurrent kernel is static and thus by default is set to not trainable when initializing the layer. If initializing using from_weights, the recurrent_nontrainable_boolean_mask is unused by default.

Parameters:

  • dt (float) – Duration of one time step

  • tau_decay (float) – Synaptic decay time

  • tau_rise (float) – Synaptic rise time

  • tau_syn (float) – Synaptic time constant

  • v_peak (float) – Voltage peak

  • v_reset (float) – Reset potential

  • I_bias (float) – Constant background current set near or at the rheobase value

  • G (float) – Scales the static weight matrix

  • Q (float) – Scales the feedback weight matrix

  • activation (str or function) – Activation function. Can be a string (i.e. ‘tanh’) or a function. This is unused by default. (Default: None)

  • initial_h (Tensor[2D float] or None) – An optional 1 x self.units tensor or numpy array specifying the initial neuron firing rates. (Default: None)

  • initial_voltage (Tensor[2D float] or None) – An optional 1 x self.units tensor or numpy array specifying the initial voltage. (Default: None)

  • kwargs – See FORCELayer for additional required args. The recurrent kernel must be static and therefore the recurrent_kernel_trainable parameter is not supported.

States:

  • t_step - 1 x 1 tensor that tracks number of time steps

  • v - 1 x self.units tensor containing voltage traces of each neuron

  • u - 1 x self.units tensor, auxillary storage variable (may be unused)

  • h - 1 x self.units tensor containing neuron firing rates (r in paper)

  • hr - 1 x self.units tensor, storage variable for double exponential filter (h in paper)

  • ipsc - 1 x self.units tensor, Post synaptic current storage variable

  • hr_ipsc - 1 x self.units tensor, storage variable for ipsc

  • out - 1 x self.output_size tensor containing predicted output

call(inputs, states)

Implements forward pass of the layer.

Parameters:

  • inputs (Tensor[2D float]) – Input tensor of shape (1, input dimensions).

  • states (list[Tensor[2D float]]) – List of tensors corresponding to the states of the layer.

Returns:

  • output (Tensor[2D float]) - 1 x self.output_size tensor containing the output of the forward pass.

  • updated states (list[Tensor[2D float]]) - List of tensors containing the updated states of the layer.

compute_current(inputs, states)

Computes current of each neuron

Parameters:

states (tuple[Tensor[2D]]) – A tuple of tensors containing the layer states

Returns:

(Tensor[2D float]) - 1 x self.units tensor of neuron currents

get_initial_state(inputs=None, batch_size=None, dtype=None)

Initializes the states of the layer (called implicitly during layer build). See: https://www.tensorflow.org/api_docs/python/tf/keras/layers/RNN

Returns:

A list of tensors containing the initial layer states

initialize_feedback_kernel(feedback_kernel=None)

Initializes the feedback kernel. Called in the layer object’s build and from_weights methods.

Parameters:

feedback_kernel (Tensor[2D float]) – Tensor array containing the pre-initialized kernel. If none, the kernel will be randomly initialized. (Default: None)

initialize_output_kernel(output_kernel=None)

Initializes the output kernel. Called in the layer object’s build and from_weights methods.

Parameters:

output_kernel (Tensor[2D float]) – Tensor or numpy array containing the pre-initialized kernel. If none, the kernel will be randomly initialized. (Default: None)

initialize_recurrent_kernel(recurrent_kernel=None)

Initializes the recurrent kernel. Called in the layer object’s build and from_weights methods.

Parameters:

recurrent_kernel (Tensor[2D float]) – Tensor containing the pre-initialized kernel. If none, the kernel will be randomly initialized. (Default: None)

initialize_voltage(batch_size)

Initializes voltage trace for each neuron

Parameters:

batch_size (int) – The batch size (Note: for potential future batched processing support. Presently, input batch size will always be one.)

Returns:

(Tensor[2D float]) A batch_size x self.units tensor of initial voltages

property state_size

size(s) of state(s) used by this cell.

It can be represented by an Integer, a TensorShape or a tuple of Integers or TensorShapes.

update_firing_rate(v_mask, states)

Updates the firing rate of each neuron

Parameters:

  • v_mask (Tensor[2D float]) – 1 x self.units zero-one mask where one indicates that the neuron voltage exceeded self.v_peak

  • states (tuple[Tensor[2D float]]) – A tuple of tensors containing the layer states

Returns:

  • h (Tensor[2D float]) - 1 x self.units tensor, updated h state

  • hr (Tensor[2D float]) - 1 x self.units tensor, updated hr state

  • ipsc (Tensor[2D float]) - 1 x `self.units tensor, updated ipsc state

  • hr_ipsc (Tensor[2D float]) - 1 x self.units tensor, updated hr_ipsc state

update_voltage(I, states)

Updates the voltage of each neuron

Parameters:

  • I (Tensor[2D float]) – 1 x self.units tensor of neuron currents

  • states (tuple[Tensor[2D float]]) – A tuple of tensors containing the layer states

Returns:

  • v (Tensor[2D float]) - 1 x self.units tensor, updated v state

  • u (Tensor[2D float]) - 1 x self.units tensor, updated u state

  • v_mask (Tensor[2D float]) - 1 x self.units zero-one mask where one indicates that the neuron voltage exceeded self.v_peak

class tension.spiking.OptimizedSpikingNN(*args, **kwargs)

Optimizations added to SpikingNN for improved computational speed. Subclass from this class when creating new spiking layers.

compute_current(inputs, states)

Computes current of each neuron

Parameters:

states (tuple[Tensor[2D]]) – A tuple of tensors containing the layer states

Returns:

(Tensor[2D float]) - 1 x self.units tensor of neuron currents

update_firing_rate(v_mask, states)

Updates the firing rate of each neuron

Parameters:

  • v_mask (Tensor[2D float]) – 1 x self.units zero-one mask where one indicates that the neuron voltage exceeded self.v_peak

  • states (tuple[Tensor[2D float]]) – A tuple of tensors containing the layer states

Returns:

  • h (Tensor[2D float]) - 1 x self.units tensor, updated h state

  • hr (Tensor[2D float]) - 1 x self.units tensor, updated hr state

  • ipsc (Tensor[2D float]) - 1 x `self.units tensor, updated ipsc state

  • hr_ipsc (Tensor[2D float]) - 1 x self.units tensor, updated hr_ipsc state

class tension.spiking.LIF(*args, **kwargs)

Leaky integrate and fire neuron

Parameters:

  • tau_mem (float) – Membrane time constant

  • tau_ref (float) – Refractory time constant

  • kwargs – See spiking.SpikingNN for additional args

initialize_voltage(batch_size)

Initializes voltage trace for each neuron

Parameters:

batch_size (int) – The batch size (Note: for potential future batched processing support. Presently, input batch size will always be one.)

Returns:

(Tensor[2D float]) A batch_size x self.units tensor of initial voltages

update_voltage(I, states)

Updates the voltage of each neuron

Parameters:

  • I (Tensor[2D float]) – 1 x self.units tensor of neuron currents

  • states (tuple[Tensor[2D float]]) – A tuple of tensors containing the layer states

Returns:

  • v (Tensor[2D float]) - 1 x self.units tensor, updated v state

  • u (Tensor[2D float]) - 1 x self.units tensor, updated u state

  • v_mask (Tensor[2D float]) - 1 x self.units zero-one mask where one indicates that the neuron voltage exceeded self.v_peak

class tension.spiking.Izhikevich(*args, **kwargs)

Implements Izhikevich neuron

Parameters:

  • adapt_time_inv (float) – Reciprocal of the adaptation time constant

  • resonance_param (float) – Controls resonance properties of the model

  • capacitance (float) – Membrane capacitance

  • adapt_jump_curr (float) – Adaptation jump current

  • gain_on_v (float) – Controls action potential half-width

  • v_resting (float) – Resting membrane potential

  • v_thres (float) – Threshold membrane potential

  • kwargs – See spiking.SpikingNN for additional args

initialize_voltage(batch_size)

Initializes voltage trace for each neuron

Parameters:

batch_size (int) – The batch size (Note: for potential future batched processing support. Presently, input batch size will always be one.)

Returns:

(Tensor[2D float]) A batch_size x self.units tensor of initial voltages

update_voltage(I, states)

Updates the voltage of each neuron

Parameters:

  • I (Tensor[2D float]) – 1 x self.units tensor of neuron currents

  • states (tuple[Tensor[2D float]]) – A tuple of tensors containing the layer states

Returns:

  • v (Tensor[2D float]) - 1 x self.units tensor, updated v state

  • u (Tensor[2D float]) - 1 x self.units tensor, updated u state

  • v_mask (Tensor[2D float]) - 1 x self.units zero-one mask where one indicates that the neuron voltage exceeded self.v_peak

class tension.spiking.Theta(*args, **kwargs)

Implements the Theta neuron. See spiking.SpikingNN for states and args.

initialize_voltage(batch_size)

Initializes voltage trace for each neuron

Parameters:

batch_size (int) – The batch size (Note: for potential future batched processing support. Presently, input batch size will always be one.)

Returns:

(Tensor[2D float]) A batch_size x self.units tensor of initial voltages

update_voltage(I, states)

Updates the voltage of each neuron

Parameters:

  • I (Tensor[2D float]) – 1 x self.units tensor of neuron currents

  • states (tuple[Tensor[2D float]]) – A tuple of tensors containing the layer states

Returns:

  • v (Tensor[2D float]) - 1 x self.units tensor, updated v state

  • u (Tensor[2D float]) - 1 x self.units tensor, updated u state

  • v_mask (Tensor[2D float]) - 1 x self.units zero-one mask where one indicates that the neuron voltage exceeded self.v_peak