In a medium bowl, whisk together white rice flour, sweet rice flour, cornstarch, baking powder, cinnamon, ginger, baking soda, xanthan gum, salt, nutmeg, and allspice. Set aside.
Add half the dry ingredients. Turn on mixer to medium speed. Blend until combined. Batter will be thick. Add pumpkin and buttermilk. Mix until combined.
Add remaining dry ingredients. Mix until a thick, fluffy batter forms. If using, stir in raisins and chopped nuts.