JEE Main · 2020 · 04 Sep 2020 · Shift 1 · Matrices

For infinitely many solutions the coefficient determinant must vanish: $\left|\begin{array}{ccc}1& -2& 3\\ 2& 1& 1\\ 1& -7& a\end{array}\right|=0$. Expanding, $1(a+7)+2(2a-1)+3(-14-1)=0$, i.e. $5a-40=0$, so $a=8$. Consistency then requires the system to be dependent, which forces the third equation to be a combination of the first two; solving for the constant gives $27-b=24$, so $b=3$. Hence $a-b=8-3=5$.