To provide a rubbery polymer (latex) having moderately increased particle diameter and exhibiting high impact resistance and designing flexibility and a latex of a polymer having an acid group, containing little small-diameter rubber latex having the original diameter and enabling the production of the above rubbery polymer.

The latex of a polymer containing an acid group is produced by combining (A) 10-99 pts.wt. of a polymer or copolymer composed of 5-100 wt.% monomer unit containing an acid group, 0-95 wt.% alkyl (meth)acrylate unit and 0-50 wt.% other copolymerizable monomer unit in the presence of (B) 1-90 pts.wt. (A+B is 100 pts.wt.) of a polyorganosiloxane.