We present a quantum impurity solver based on a pseudoparticle framework, which combines diagrammatic resummations for a three-point vertex with diagrammatic Monte Carlo sampling of a four-point vertex. This recently proposed approach (A. J. Kim, arXiv:2112.15549) is generalized here to fermionic impurity problems and we discuss the technical details of the implementation, including the time-stepping approach, the Monte Carlo updates, and the routines for checking the two-particle irreducibility of the four-point vertex. We also explain how the vertex information can be efficiently stored using a Dubiner basis representation. The convergence properties of the algorithm are demonstrated with applications to exactly solvable impurity models and dynamical mean field theory simulations of the single-orbital Hubbard model. It is furthermore shown that the algorithm can handle a two-orbital problem with off-diagonal hybridizations, which would cause a severe sign problem in standard hybridization-expansion Monte Carlo simulations. Since the vertex-based algorithm successfully handles sign-oscillating integrals in equilibrium and samples only connected diagrams, it may be a promising approach for real-time simulations.