Feature Interaction·Huawei·2017Ranking

DeepFMFactorization-Machine based Neural Network

Replace Wide & Deep's hand-made crosses with a Factorization Machine that shares embeddings with the deep tower.

DeepFM: A Factorization-Machine based Neural Network for CTR Prediction

§1The key contribution

Replace hand-engineered crosses with a factorization machine that shares its embeddings with the deep network.

The problem

Wide & Deep still needs humans to design the cross features for its wide part, and the crosses do not generalize to pairs never seen together.

The idea

Use an FM as the 'wide' part. It models every pairwise interaction as an inner product of field embeddings, and those same embeddings feed the MLP. Low- and high-order interactions are learned end-to-end from raw features.

Raw inputEmbedding / tokenizerInteraction / operatorMLP / linearOutput / loss click any block

Figure 1. DeepFM architecture. Arrows show data flow; annotations show tensor shapes or symbols. Click a block for details, or use the walkthrough to step through the forward pass.

§2Breaking it down

The contribution, piece by piece. Select a card to highlight the blocks it refers to in Figure 1.

§3How it works

DeepFM keeps the two-branch layout of Wide & Deep but makes the wide part learnable. A Factorization Machine models every pairwise interaction as the inner product of two field embeddings, ⟨eᵢ, eⱼ⟩. The same embeddings also feed an MLP. Because the embeddings are shared, the low-order (FM) and high-order (DNN) signals train the same representation, and no feature engineering is needed.

Lineage. It combines FM (Rendle, 2010) with Wide & Deep. Related designs: NFM and xDeepFM (CIN for vector-wise high-order crosses).

§4Key equations

FM component

y_{FM} = \langle w, x \rangle + \sum_{i=1}^{m}\sum_{j=i+1}^{m} \langle V_i, V_j \rangle\, x_i\, x_j

Linear-time pairwise trick

\sum_{i<j} \langle e_i, e_j\rangle = \tfrac12 \sum_{f=1}^{k} \Big[ \big(\textstyle\sum_i e_{i,f}\big)^2 - \sum_i e_{i,f}^2 \Big]

Output

\hat y = \sigma(y_{FM} + y_{DNN})

§5Why it works

Inner-product interactions generalize to pairs never seen together, because they reuse each field's embedding.
Sharing embeddings means the FM term acts as a strong inductive bias that also shapes the deep tower's input.

§6Limitations & trade-offs

Explicit interactions stop at 2nd order. Higher orders are left to the MLP, which is inefficient at multiplicative patterns.
Every pair has the same form ⟨eᵢ,eⱼ⟩, with no per-pair weighting. FwFM, AFM and FiBiNET add that.