In this paper, a broadband vibrational energy harvester (VEH) is developed based on a soft magneto-sensitive elastomer (SMSE). The utilization of SMSE provides the VEH with a strong softening effect when it is subjected to the magnetic field of a permanent magnet. A polyvinylidene fluoride (PVDF) layer is attached to the SMSE to convert vibrational energy into electricity. A magneto-electro-mechanical model based on finite element method is developed and is further reduced using assumed modes. Two prototypes are fabricated with different SMSE thicknesses (3 mm and 5 mm respectively). Frequency sweep experiments are conducted to investigate their broadband behavior. The difference between jumping frequencies are 2.39 and 4.34 Hz at a low acceleration level of 0.3 g. With the broadband characteristics of SMSE, the two prototypes are capable of providing power over a wide frequency range and generating an average power of 0.096 μW and 0.11 μW respectively with a load of 4.7 MΩ.