Key Insight: Each row has exactly one $1$ and rest $0$s. Each column also has exactly one $1$ and rest $0$s.

So no two rows can have $1$ in the same column. This is exactly a Permutation Matrix!

Count choices row by row

Row 1: $5$ columns available $\rightarrow$ 5 choices
Row 2: $4$ columns remaining $\rightarrow$ 4 choices
Row 3: $3$ columns remaining $\rightarrow$ 3 choices
Row 4: $2$ columns remaining $\rightarrow$ 2 choices
Row 5: $1$ column remaining $\rightarrow$ 1 choice

Total number of matrices

$\text{Total} = 5 \times 4 \times 3 \times 2 \times 1 = 5! = 120$

Answer: $\boxed{120}$