This paper presents a new methodology for cost optimization of the preliminary layout design of three‐dimensional reinforced concrete (RC) frames. This approach is capable of being easily employed for the optimal layout design of a realistic large RC structure that accounts for constraints imposed by design standards. The new approach considers modeling, structural analysis, concrete member design, and discrete optimization together with data on cost of systems and materials. The methodology comprises two parts. First, using the cross‐sectional action effects as design variables, a heuristic cost function is presented as an alternative to traditional cost functions for layout optimization of RC structures. Using the presented cost function, a structural optimization problem is formulated for column layout design of 3D RC frames. Then, an Ant System (AS) algorithm, a discrete method, is proposed to solve the cost optimization problem. Two comparative design examples are included to demonstrate the performance of the new methodology and the excellent convergence of the Ant Colony Optimization (ACO) algorithm.