JEE Main · 2021 · 17 Mar 2021 · Shift 1 · Inverse Trigonometric Functions

Taking the tangent of both sides,

which reduces to $4x^2+31x-8=0$ with roots $x=\frac{1}{4}$ and $x=-8$. Both are smaller than $1$, so $\frac{1}{x-1}<0$ and ${\cot }^{-1}\frac{1}{x-1}=\pi +{\tan }^{-1}(x-1)$; each root must therefore be put back into the original equation.
For $x=\frac{1}{4}$: $(x+1)(x-1)=-\frac{15}{16}<1$, so ${\tan }^{-1}\frac{5}{4}+{\tan }^{-1}\left(-\frac{3}{4}\right)={\tan }^{-1}\frac{8}{31}$ and the left side equals $\pi +{\tan }^{-1}\frac{8}{31}$, which is not ${\tan }^{-1}\frac{8}{31}$. This root is rejected.
For $x=-8$: ${\tan }^{-1}(-7)$ and ${\tan }^{-1}(-9)$ are both negative and $(-7)(-9)=63>1$, so ${\tan }^{-1}(-7)+{\tan }^{-1}(-9)=-\pi +{\tan }^{-1}\frac{8}{31}$ and the left side equals ${\tan }^{-1}\frac{8}{31}$. This root is valid.
The only possible value is $x=-8$, so the sum of the possible values is $-\frac{32}{4}$.