Ezri Konsa’s brace cancelled out Harvey Barnes’ opener as Villa moved up to 10th in the Premier League on Sunday.
They have won three of Gerrard’s four games in charge since he replaced Dean Smith last month and the former Rangers boss expects them to continue to improve.
“We have come in on the back of five defeats, the players could have been lacking confidence and belief but the reaction we have had has been terrific,” he said, after victory over his former Liverpool manager Brendan Rodgers.
“We can be better. The important thing is to try to stay there and improve the position we are in. We want to be bold and keep pushing for every point that’s available.
“The two games we played this week, both our second-half performances have been top and there’s lot to be proud and pleased of.
“We had to get into them a little bit and demand a reaction [at the break], I didn’t think we were at our best. There are still a lot of teething problems to iron out but I’m delighted with the win. It’s a big start for us.”
Leicester took a 14th-minute lead when Patson Daka – in for the benched Jamie Vardy – shook off lightweight Villa challenges to find Barnes and he coolly found the bottom corner.
Villa, though, levelled three minutes later when Jonny Evans failed to clear Douglas Luiz’s free-kick and Matty Cash returned the ball back into the centre.
Emi Buendia met it and his header was dropping in before Konsa got the slightest final touch.
Villa thought they had grabbed the lead just as the game entered first-half stoppage time only to be denied, and frustrated, by VAR.
Schmeichel stretched to claim Cash’s header only for Ramsey to smash in from close range with the goalkeeper having briefly trapped the ball between the ground and his hand.
VAR official Paul Tierney advised referee Michael Oliver to check whether Schmeichel had control of the ball and the goal was ruled out as the decision went in Leicester’s favour.
But it was the trigger for Villa to channel their anger to victory and, after Luke Thomas blocked Ollie Watkins’ drive, the hosts grabbed the winner after 54 minutes.
John McGinn’s corner found Konsa at the far post and he got above Caglar Soyuncu to squeeze the ball between Schmeichel and the upright.
Leicester struggled to live with Villa’s aggression and Schmeichel denied Watkins before Emi Martinez kept the points safe with 15 minutes left.
Barnes met Timothy Castagne’s centre with a looping header which seemed bound for the top corner only for Martinez to brilliantly turn the ball over.
Defeat left Leicester 11th in the table and they have now conceded 10 set-piece goals, a joint league high with Crystal Palace.
Boss Rodgers said: “We’re frustrated, I thought in the first half we were very good and could have had more goals.
“Our Achilles heel which has killed us is our defending of the set-piece. The situation can sap the confidence out of the team.
“The only threat against us in the first half was when the ball went dead. In the second half we didn’t start so fast, that happens sometimes, but the corner we conceded from was hugely disappointing. It’s an area we spoke about before the game.
“We don’t get tight enough on the marking and he’s able to squeeze it in the far post. These are undoing some really good parts of our game. It’s something we’ve got to fix.”