We investigate several optimizations to a recentlypublished architecture for extracting bottleneck features forlarge-vocabulary speech recognition with deep neural networks.We are able to improve recognition performance of first-passsystems from a 12% relative word error rate reduction reportedpreviously to 21%, compared to MFCC baselines on a Tagalogconversational telephone speech corpus. This is achieved byusing different input features, training the network to predictcontext-dependent targets, employing an efficient learning rateschedule and varying several architectural details. Evaluationson two larger German and French speech transcription tasksshow that the optimizations proposed are universally applicableand yield comparable gains on other corpora (19.9% and 22.8%,respectively)