Hundreds of thousands of people have been displaced over the past year by violence there.
Local media reported that the grave was found during a probe into alleged atrocities committed by the former president of Ethiopia's Somali region.
Abdi Mohammed is awaiting trial over allegations he fuelled ethnic clashes.
A notorious regional security force known as the Liyu police is accused of carrying out killings in the area between the Somali and Oromia regions and it reported directly to the regional president.
The police are trying to identify the 200 bodies found.
Mr Mohammed was forced to resign in August and was arrested weeks later after violence broke out in the regional capital, Jijiga.
He is alleged to have overseen widespread rights abuses including torture, rape and killings during his 13-year rule.
Last month the state-linked broadcaster Fana reported that he attempted to escape police custody by climbing through a window ahead of a court appearance.