ML for Scientists: L4

 

Carol(ina) Cuesta-Lazaro

This week in AI4Science

Image Credit: CS231n Convolutional Neural Networks for Visual Recognition
4

Pixel 1

Pixel 2

Pixel N

Multilayer Perceptron (MLP)

a^{(l)} = f^{(l)}(W^{(l)}a^{(l-1)} + b^{(l)})

Non-Linearity

Weights

Biases

Architectures

"Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges" Bronstein et al

Symmetries as Inductive Biases

=
p(
)
p(
p(
)

Invariant

Equivariant

=
p(
)
p(
=
p(
)
p(

All learnable functions

All learnable functions constrained by your data

All Equivariant functions

More data efficient!

Inductive bias: Translation Invariance

Data Representation: Images

Image Credit: Irhum Shakfat "Intuitively Understanding Convolutions for Deep Learning"

Convolutional Neural Networks (CNNs)

Inductive bias: Permutation Invariance

Data Representation: Sets, Pointclouds

+
+
= 4
f(x) = f(P(x))
f(x) = \oplus_{i=0}^N h_\theta(x_i)
+
+
= 4

Deep Sets

Inductive bias: Permutation Invariance

Data Representation: Graph

Graph Neural Networks

x_i
x_1
x_2
x_j
m_{ij} = f_e(x_i, x_j, e_{ij})

Edge:

h_{i} = f_n(x_i, \mathcal{A}_j e_{ij})

Node:

Message

Node features

{Galaxy Luminosity}

Edge features

{Distance}

Edge Predictions

{Force of j on i}

Node embeddings

Aggregator

{Max, Mean, Variance...}

Permutation Invariant

Node Predictions

{Galaxy Peculiar Velocity}

Graph Predictions

{Cosmological Parameters}

f_g(\mathcal{A}_i h_i)

Transformers might be the unifying architecture!

 

Text

Images

Transformers

Data Representation: Sets, Pointclouds, Sequences, Images...

Inductive bias: Permutation Invariance

"The dog chased the cat because it was playful."

But, we decide to break permutation invariance!

"Dog bites man" !=

"Man bites dog"

PE_{(\text{pos}, 2i)} = \sin\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)
PE_{(\text{pos}, 2i+1)} = \cos\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)

Unique encoding per position (regardless of sequence length)

Easty to compute "distances": pos -> pos + diff

Generalizes to longer sequences than used for training

Wish List for Encoding Positions:

The bitter lesson by Rich Sutton

The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. [...]

 

methods that continue to scale with increased computation even as the available computation becomes very great. [...]

 

We want AI agents that can discover like we can, not which contain what we have discovered.​