To provide a cellulose ester-based composite yarn that is capable of imparting an excellent dry touch feeling, a glossy feeling, and a proper swelling feeling and an excellent elasticity to a woven or knitted fabric, and a woven or knitted fabric having such features.

The cellulose ester-based composite yarn 11 is prepared by combining a polyester crimped yarn 8 obtained by false-twist processing a polyester highly-oriented undrawn yarn 1 with a condition in which a false twist coefficient is 5.0×10³-1.5×10⁴, a false-twist temperature is (Tg+40°C) to (mp-30°C) (Tg: the glass-transition temperature of a polymer (°C), mp: the melting point of a polymer (°C)), and a twisting tension is 0.1-0.3 cN/dtex and a cellulose ester-based filament yarn 2 through a fiber-combining and entangling treatment. The woven or knitted fabric is obtained from the yarn.