A search for new physics in top quark production is performed in proton-proton collisions at 13TeV. The data set corresponds to an integrated luminosity of 35.9fb^-1 collected in 2016 with the CMS detector. Events with two opposite-sign isolated leptons (electrons or muons), and b quark jets in the final state are selected. The search is sensitive to new physics in top quark pair production and in single top quark production in association with a W boson. No significant deviation from the standard model expectation is observed. Results are interpreted in the framework of effective field theory and constraints on the relevant effective couplings are set, one at a time, using a dedicated multivariate analysis. This analysis differs from previous searches for new physics in the top quark sector by explicitly separating tW from tt¯ events and exploiting the specific sensitivity of the tW process to new physics.