Purpose: Novel radiotherapy techniques like synchrotron X-ray microbeam radiation therapy (MRT) require fast dose distribution predictions that are accurate at the sub-mm level, especially close to tissue/bone/air interfaces. Monte Carlo (MC) physics simulations are recognized to be one of the most accurate tools to predict the dose delivered in a target tissue but can be very time consuming and therefore prohibitive for treatment planning. Faster dose prediction algorithms are usually developed for clinically deployed treatments only. In this work, we explore a new approach for fast and accurate dose estimations suitable for novel treatments using digital phantoms used in preclinical development and modern machine learning techniques. We develop a generative adversarial network (GAN) model, which is able to emulate the equivalent Geant4 MC simulation with adequate accuracy and use it to predict the radiation dose delivered by a broad synchrotron beam to various phantoms. Methods: The energy depositions used for the training of the GAN are obtained using full Geant4 MC simulations of a synchrotron radiation broad beam passing through the phantoms. The energy deposition is scored and predicted in voxel matrices of size 140 × 18 × 18 with a voxel edge length of 1 mm. The GAN model consists of two competing 3D convolutional neural networks, which are conditioned on the photon beam and phantom properties. The generator network has a U-Net structure and is designed to predict the energy depositions of the photon beam inside three phantoms of variable geometry with increasing complexity. The critic network is a relatively simple convolutional network, which is trained to distinguish energy depositions predicted by the generator from the ones obtained with the full MC simulation. Results: The energy deposition predictions inside all phantom geometries under investigation show deviations of less than 3% of the maximum deposited energy from the simulation for roughly 99% of the voxels in the field of the beam. Inside the most realistic phantom, a simple pediatric head, the model predictions deviate by less than 1% of the maximal energy deposition from the simulations in more than 96% of the in-field voxels. For all three phantoms, the model generalizes the energy deposition predictions well to phantom geometries, which have not been used for training the model but are interpolations of the training data in multiple dimensions. The computing time for a single prediction is reduced from several hundred hours using Geant4 simulation to less than a second using the GAN model. Conclusions: The proposed GAN model predicts dose distributions inside unknown phantoms with only small deviations from the full MC simulation with computations times of less than a second. It demonstrates good interpolation ability to unseen but similar phantom geometries and is flexible enough to be trained on data with different radiation scenarios without the need for optimization of the model parameter. This proof-of-concept encourages to apply and further develop the model for the use in MRT treatment planning, which requires fast and accurate predictions with sub-mm resolutions.