Introduction Oral formulations of docetaxel have successfully been developed as an alternative for intravenous administration. Co-administration with the enzyme inhibitor ritonavir boosts the docetaxel plasma exposure. In dose-escalation trials, the maximum tolerated doses for two different dosing regimens were established and dose-limiting toxicities (DLTs) were recorded. The aim of current analysis was to develop a pharmacokinetic (PK)-toxicodynamic (TOX) model to quantify the relationship between docetaxel plasma exposure and DLTs. Methods A total of 85 patients was included in the current analysis, 18 patients showed a DLT in the four-week observation period. A PK-TOX model was developed and simulations based on the PK-TOX model were performed. Results The final PK-TOX model was characterized by an effect compartment representing the toxic effect of docetaxel, which was linked to the probability of developing a DLT. Simulations of once-weekly, once-daily 60 mg and once-weekly, twice-daily 30 mg followed by 20 mg of oral docetaxel suggested that 14 % and 34 % of patients, respectively, would have a probability >25 % to develop a DLT in a four-week period. Conclusions A PK-TOX model was successfully developed. This model can be used to evaluate the probability of developing a DLT following treatment with oral docetaxel and ritonavir in different dosing regimens.