I believe the main reason was safety. Ground effects were allowing cars to take corners at frightening speeds, and coupled with low safety measures at the time were resulting in too many deaths. The technology surrounding GE had high potential and cars were simply going faster and faster until they decided to slow them down.
The safety is much improved today which is why there are many asking for it again.
This, plus if you hit something like another car and got launched into the air, the sudden loss of ground effect meant the cars flew quite a height and distance before they crashed back into the ground. As evidenced by Villeneuve's fatal crash at Zolder and Pironi's career ending crash at Hockenheim.
It was these incidents that effectively brought about the end of ground effect in F1.
Also, as the cars had to have extremely stiff suspension to cope with the downforce, drivers would get battered just trying to drive the things through the corners.

"I knew I had been beaten by the best driver in the world." Rene Arnoux.