LOX/LH2 subscale rocket nozzle flow fields are computationally simulated using the three-dimensional compressible Navier-Stokes equations. The area ratio of the nozzle is 140 and film coolant hydrogen gases are injected from 30 film cooling holes which are distributed circumferentially at the area ratio of 13. The experimental nozzle throat Reynolds number indicates that the boundary layer of the nozzle is in its transition region as the size of the nozzle is small. Clear difference in effective specific impulses of the secondary flow between the laminar and turbulent conditions is also shown. The nozzle wall temperature also influences on the nozzle performance and the experimental performances were in better agreement with the laminar computations when the wall temperature is set to 300K which is closer to the experimental conditions. Both the turbulent and laminar computations are carried out to investigate the effect of the boundary layer conditions to the nozzle performance. The computed results show that the structure of the separated flow down stream of the film cooling injection significantly changes between the turbulent and laminar conditions.