The correct simple sentence is – Although he spoke clumsily, I listened to him very carefully.

Though his speech was clumsy, I listened to him carefully.

Despite the clumsiness of his speech, I listened to him carefully.

He spoke clumsily, then I listened to him carefully.

Despite speaking clumsy manner, I listened to him carefully.