This paper investigates the possibility to improve the filtering process of flue gas by separation of suspended nanoparticle using dielectrophoresis. The study focuses on the particles having an average radius of about 50-150 nm, that cannot be filtrated by classical techniques but have a harmful effect for environment and human health. The size distribution nanoparticles collected from the flue gas filters of a hazardous waste incinerator plant were evaluated. Based on obtained experimental data and a proposed mathematical model, the concentration distribution of nanoparticle suspended in flue gas inside a microfluidic separation device was analyzed by numerical simulations, using the finite element method. The performances of the device were described in terms of three new specific quantities related to the separation process, namely Recovery, Purity and Separation Efficiency. The simulations could provide the optimal values of control parameters for separation process, and aim to be a useful tool in designing microfluidic devices for separating nanoparticle from combustion gases.