ML for Scientists: L4
Carol(ina) Cuesta-Lazaro
This week in AI4Science
Image Credit: CS231n Convolutional Neural Networks for Visual Recognition
4
Pixel 1
Pixel 2
Pixel N
Multilayer Perceptron (MLP)
a^{(l)} = f^{(l)}(W^{(l)}a^{(l-1)} + b^{(l)})
Non-Linearity
Weights
Biases
Architectures
Symmetries as Inductive Biases
=
p(
)
p(
p(
)
Invariant
Equivariant
=
p(
)
p(
=
p(
)
p(
All learnable functions
All learnable functions constrained by your data
All Equivariant functions
More data efficient!
Inductive bias: Translation Invariance
Data Representation: Images
Image Credit: Irhum Shakfat "Intuitively Understanding Convolutions for Deep Learning" Convolutional Neural Networks (CNNs)
Inductive bias: Permutation Invariance
Data Representation: Sets, Pointclouds
+
+
= 4
f(x) = f(P(x))
f(x) = \oplus_{i=0}^N h_\theta(x_i)
+
+
= 4
Deep Sets
Inductive bias: Permutation Invariance
Data Representation: Graph
Graph Neural Networks
x_i
x_1
x_2
x_j
m_{ij} = f_e(x_i, x_j, e_{ij})
Edge:
h_{i} = f_n(x_i, \mathcal{A}_j e_{ij})
Node:
Message
Node features
{Galaxy Luminosity}
Edge features
{Distance}
Edge Predictions
{Force of j on i}
Node embeddings
Aggregator
{Max, Mean, Variance...}
Permutation Invariant
Node Predictions
{Galaxy Peculiar Velocity}
Graph Predictions
{Cosmological Parameters}
f_g(\mathcal{A}_i h_i)
Transformers might be the unifying architecture!
Text
Images
Transformers
Data Representation: Sets, Pointclouds, Sequences, Images...
Inductive bias: Permutation Invariance
"The dog chased the cat because it was playful."
But, we decide to break permutation invariance!
"Dog bites man" !=
"Man bites dog"
PE_{(\text{pos}, 2i)} = \sin\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)
PE_{(\text{pos}, 2i+1)} = \cos\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)
Unique encoding per position (regardless of sequence length)
Easty to compute "distances": pos -> pos + diff
Generalizes to longer sequences than used for training
Wish List for Encoding Positions:
The bitter lesson by Rich Sutton
The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. [...]
methods that continue to scale with increased computation even as the available computation becomes very great. [...]
We want AI agents that can discover like we can, not which contain what we have discovered.
ML For Scientists: L4
By carol cuesta
