JEE Main · 2005 · 08 May 2005 · Offline · Binomial Theorem

In $(a{x}^2+\frac{1}{bx}{)}^{11}$ the general term is $\left(\genfrac{}{}{0ex}{}{11}{r}\right)a^{11-r}{b}^{-r}{x}^{22-3r}$; the term in $x^7$ requires $22-3r=7$, so $r=5$, with coefficient $\left(\genfrac{}{}{0ex}{}{11}{5}\right)a^6{b}^{-5}$. In $(ax-\frac{1}{b{x}^2}{)}^{11}$ the general term is $\left(\genfrac{}{}{0ex}{}{11}{r}\right)a^{11-r}(-1{)}^r{b}^{-r}{x}^{11-3r}$; the term in $x^{-7}$ requires $11-3r=-7$, so $r=6$, with coefficient $\left(\genfrac{}{}{0ex}{}{11}{6}\right)a^5{b}^{-6}$. Equating the two and using $\left(\genfrac{}{}{0ex}{}{11}{5}\right)=\left(\genfrac{}{}{0ex}{}{11}{6}\right)$ gives $a^6{b}^{-5}=a^5{b}^{-6}$, hence $ab=1$.