Multiply features to express interactions

Feature interactions

Suppose a prediction depends on the combination of two features, not just their separate effects. A cross layer builds those products explicitly. Input x [B, D] has B examples and D numerical features per example; output [B, D] contains transformed features, not probabilities. Here B = 2 and D = 2. Real inputs could concatenate normalized numerical features and learned category embeddings.

View the full file · cross_layer.py

1import torch2from torch import nn3 4class CrossNetworkV1(nn.Module):5    def __init__(self, dim, depth=2):6        super().__init__()7        self.weights = nn.ParameterList([8            nn.Parameter(torch.randn(dim) * dim ** -0.5) for _ in range(depth)9        ])10        self.biases = nn.ParameterList([11            nn.Parameter(torch.zeros(dim)) for _ in range(depth)12        ])13 14    def forward(self, x0):15        # x0: [B, D]; each row is one example's combined features.16        x = x017        for weight, bias in zip(self.weights, self.biases):18            projection = (x * weight).sum(dim=-1, keepdim=True)  # [B, 1]19            x = x0 * projection + bias + x                     # [B, D]20        return x21 22if __name__ == "__main__":23    torch.manual_seed(0)24    x = torch.tensor([[2., 3.], [1., 4.]])  # [B=2, D=2]25    cross = CrossNetworkV1(dim=2, depth=1)26    with torch.no_grad():27        cross.weights[0].copy_(torch.tensor([0.5, 1.]))28        print(cross(x))  # [[10., 15.], [5.5, 22.]]

Lines 22–28
Work through one example
For input [2, 3] and weight [0.5, 1], the weighted sum is 2×0.5 + 3×1 = 4. Multiply the original [2, 3] by 4 and add the original again: [8, 12] + [2, 3] = [10, 15]. Bias starts at zero. The main section initializes the layer and sets weights explicitly so you can check the arithmetic.
One shared scalar scales every original feature
1. First layer: current x = original x0
  Input x0[2, 3]
  Learned weights w[0.5, 1]
2. Weighted sum: 2 × 0.5 + 3 × 1
  One projection4
3. Multiply the original vector by that scalar
  x0 × 4[8, 12]
4. Add bias [0, 0] and previous x [2, 3]
  Next feature vector[10, 15]
Later layers update x but keep x0 fixed. In a batch, each example gets its own scalar; all examples share the learned weights.
Python
```
20        return x21 22if __name__ == "__main__":23    torch.manual_seed(0)24    x = torch.tensor([[2., 3.], [1., 4.]])  # [B=2, D=2]25    cross = CrossNetworkV1(dim=2, depth=1)26    with torch.no_grad():27        cross.weights[0].copy_(torch.tensor([0.5, 1.]))28        print(cross(x))  # [[10., 15.], [5.5, 22.]]
```
Quick checkSame layer, weight [0.5, 1], input [1, 4]. What is the weighted sum, projection?

cross_layer.py

1import torch2from torch import nn3 4class CrossNetworkV1(nn.Module):5    def __init__(self, dim, depth=2):6        super().__init__()7        self.weights = nn.ParameterList([8            nn.Parameter(torch.randn(dim) * dim ** -0.5) for _ in range(depth)9        ])10        self.biases = nn.ParameterList([11            nn.Parameter(torch.zeros(dim)) for _ in range(depth)12        ])13 14    def forward(self, x0):15        # x0: [B, D]; each row is one example's combined features.16        x = x017        for weight, bias in zip(self.weights, self.biases):18            projection = (x * weight).sum(dim=-1, keepdim=True)  # [B, 1]19            x = x0 * projection + bias + x                     # [B, D]20        return x21 22if __name__ == "__main__":23    torch.manual_seed(0)24    x = torch.tensor([[2., 3.], [1., 4.]])  # [B=2, D=2]25    cross = CrossNetworkV1(dim=2, depth=1)26    with torch.no_grad():27        cross.weights[0].copy_(torch.tensor([0.5, 1.]))28        print(cross(x))  # [[10., 15.], [5.5, 22.]]

Work through one example