It is -us. Both endings exist in Latin genus names and both are unstressed in English, which reduces them to the same faint vowel. You cannot hear the difference and there is no rule to fall back on: Elaeagnus takes -us, Ribes takes -es, Clematis takes -is. The ending has to be learned with the name.