The black smoke was seen by thousands of visitors to St Peter's Square.
It means the 115 cardinals will hold a new round of voting tomorrow morning.
They will remain sequestered behind the Vatican's mediaeval walls until they elect a successor to Pope Benedict, who abdicated last month.
The cardinals will hold four ballots a day from tomorrow until one man has won a two-thirds majority - or 77 votes.
When the cardinals agree on a pontiff, white smoke will rise from the makeshift chimney on the chapel roof and the bells of St Peter's basilica will peal.
Earlier, the cardinals filed into the chapel praying for divine guidance in their choice of a man to succeed Pope Benedict at one of the most difficult times in the history of the Catholic Church.
With the Latin command, "Extra Omnes" (Everyone Out), the master of ceremonies ordered all those not involved in the secret ballot to leave the chapel.
The doors were then shut. A few assistants are allowed to stay inside to help but must leave each time the cardinals vote.
Dean of the College of Cardinals Angelo Sodano was the main celebrant and thanked Benedict XVI for a "brilliant pontificate".
His words were followed by a spontaneous burst of applause from the cardinals present and the congregation.
Cardinal Sodano said he wanted to renew the gratitude of the church to Benedict and prayed that through the conclave, God would "grant another good shepherd to the church".
The cardinal said it was important that the church worked "together to build upon the unity of this whole church".
There were prayers in Swahili and Portuguese and a reading in English.
Cardinals moved into Casa Santa Marta, a Vatican hotel, this morning where they will reside and eat until a new pope is elected.
The average length of the last nine conclaves was just over three days and none went on for more than five days.
As in medieval times, the cardinals are banned from communicating with the outside world.
The Vatican has also taken high-tech measures to ensure secrecy in the 21st century, including electronic jamming devices to prevent eavesdropping.
The cardinals will emerge from their seclusion only when they have chosen the 266th pontiff in the 2,000-year-history of the church.
Vatican insiders say Italy's Angelo Scola and Brazil's Odilo Scherer have emerged as the men to beat.
The former would bring the papacy back to Italy for the first time in 35 years, while the latter would be the first non-European pope in 1,300 years.
However, a host of other candidates from numerous nations have also been mentioned, including US cardinals Timothy Dolan and Sean O'Malley, Canada's Marc Ouellet and Argentina's Leonardo Sandri.
Pope Benedict XVI abdicated last month saying he was not strong enough to confront the church's woes of sex abuse scandals, bureaucratic infighting, financial difficulties and the rise of secularism.