Combine sugar, flour, and salt in a small mixing bowl. Whisk gently to break up any lumps, and set aside. Prepare an ice water bath by filling a bowl halfway with ice and water; set aside.
Combine buttermilk and egg yolks in a small saucepan and whisk untilsmooth. Whisk in dry ingredients and place the pan over medium heat. Cook, whisking constantly, until mixture is thick enough to coat the back of a spoon, about 4 minutes.
Remove from heat, and stir in vanilla extract or reserved vanilla seeds. Transfer pastry cream to a small bowl nestled in the ice water bath and let cool completely, stirring occasionally to help it cool faster.