Tokyo (CNN) -- Officials asked workers at the Fukushima Daiichi nuclear plant to evacuate temporarily Wednesday after a white cloud of smoke rose above the plant and radiation levels spiked.

Workers were "asked to withdraw to a safe area," Chief Cabinet Secretary Yukio Edano said. Authorities later allowed them to return after radiation levels dropped, the Tokyo Electric Power Company said.

Radiation levels at the plant have surged and dropped repeatedly over the past few days. The most recent spike "probably" occurred "because the containment vessel in reactor No. 3 has been damaged," a spokesman for Japan's nuclear safety agency later told reporters.

Edano said the smoke or vapor above the plant may have been caused by breach in the containment structure around the No. 3 reactor's containment vessel -- the steel and concrete shell that insulates radioactive material inside.

It was the second visible sign of trouble at the earthquake-damaged nuclear plant Wednesday.

Earlier in the day, firefighters battled a blaze in the plant's No. 4 reactor building.

Since the 9.0-magnitude earthquake and the resulting tsunami Friday, the plant has suffered a series of setbacks that have heightened fears about the possibility of widespread radiation contamination.

A hydrogen explosion took place at the plant's No. 2 reactor Tuesday. Hydrogen explosions had previously occurred in the plant's No. 1 and No. 3 reactors.

Another fire had broken out Tuesday in the No. 4 reactor. While it burned, radiation levels at the plant increased to about 167 times the average dose, the International Atomic Energy Agency said.

That dose quickly diminished with distance from the plant, and radiation fell back to levels where it posed no immediate public health threat, Chief Cabinet Secretary Yukio Edano said.

But the deteriorating situation and concerns about a potential shift in wind direction that could send radiation toward populated areas prompted authorities to warn people as far as 18.6 miles (30 kilometers) from the plant to stay inside.

"There is still a very high risk of further radioactive material coming out," Prime Minister Naoto Kan said Tuesday, asking people to remain calm.

In all, the plant holds six reactors. At the time of Friday quake, Unit 4 was shut for maintenance and all fuel from the reactor had been moved to its spent fuel pool. Units 5 and 6 were also shut at the time of the quake, but both its reactors are loaded with fuel, the IAEA said.

About 200,000 people living within a 12.4-mile (20 kilometer) radius of the plant already had been evacuated.

Authorities also banned flights over the area.

Between Units 3 and 4, Japanese authorities said they had measured radiation dose rates of up to 400 millisieverts-per-hour, IAEA reported. That's equivalent to about 2,000 chest x-rays per hour, the agency said on its website. "This is a high dose-level value, but it is a local value at a single location and at a certain point in time," it added.

As a result of the monitoring of about 150 people from around the Daiichi site, 23 have been decontaminated, IAEA said.

The number of nuclear workers who remained on site has been slashed from 800 to 50.

"Their situation is not great," said David Brenner, director of the Center for Radiological Research at Columbia University. "It's pretty clear that they will be getting very high doses of radiation. There's certainly the potential for lethal doses of radiation. They know it, and I think you have to call these people heroes."

Although the plant's three functioning reactors shut down automatically when the quake occurred, the tsunami that followed swamped the diesel generators that provided backup power to the reactor cooling systems.

Crews restored backup power, but problems keeping the reactors cool forced plant officials to take the drastic step of flooding them with seawater. Still, pressure buildups, problems with valves and a failure to fill a generator's gas tank led to hydrogen explosions and other problems.

Tuesday's events appeared to escalate the situation: Edano said the radiation releases from the explosion and fire were the first that appeared to pose a threat to human health, if only briefly.

On Monday, an explosion in the building housing the plant's No. 3 reactor apparently damaged both a water-filled chamber at the base of the reactor and the reactor containment unit itself, IAEA Director General Yukiya Amano told reporters Tuesday.

Damage to the core involved about 5% of its nuclear fuel, Amano said.

It was unclear how much radioactive material may have been emitted, what kind of health threat that could pose or when the danger would end.

"There are enormous quantities of radiation," said Dr. Ira Helfand, a member of Physicians for Social Responsibility, which opposes the use of nuclear power. "The containment is not nearly as good as around the reactor cores themselves. The potential for a major release of radiation from those sites is very real, as we saw last night with the fire."

Japanese officials told the IAEA that radioactivity had been released "directly into the atmosphere" during Tuesday's fire in the No. 4 reactor, the U.N. watchdog organization said.

Crews put that fire out, and by Tuesday afternoon, Edano said, radiation readings -- which had reached dangerously high levels at the plant earlier -- had decreased.

Still, concerns about radioactive fuel boiling off its coolant and igniting continued Wednesday. Plant operators and government officials initially considered using helicopters to drop water into the cooling pond through the damaged roof of the reactor building, but rejected the idea when they discovered that the spent fuel pond was too far from the hole in the roof, a Kyodo News report said.

In addition, Edano said, cooling systems at two other reactors, No. 5 and No. 6, were "not functioning well."

Plant managers were considering removing panels from the buildings housing those reactors in an effort to prevent the hydrogen buildup that officials believe caused the other explosions, the IAEA said.

Edano said earlier that he could not rule out the possibility of a meltdown at all three troubled reactors at the plant.

A meltdown occurs when nuclear fuel rods cannot be cooled and melt the steel and concrete structure containing them. In the worst-case scenario, the fuel can spill out of the containment unit and spread radioactivity through the air and water. That, public health officials say, can cause both immediate and long-term health problems, including radiation poisoning and cancer.

If fuel rods inside the reactors are melting, "the million-dollar question is whether that melting will be contained," said James Walsh, a CNN contributor and research associate at the Massachusetts Institute of Technology's security studies program.

At present, the long-term impact on public health from the crisis appears minimal, Brenner said.

"I think, at this point in time, there's no real evidence that there are health risks to the general population," he said.

The weather has emerged as a key variable, but on Wednesday morning, winds were blowing out to sea, CNN Meteorologist Sean Morris said.

Radiation levels in Tokyo, about 225 kilometers (140 miles) southwest of the plant, were twice the usual level on Tuesday. But the concentration -- 0.809 microsieverts per hour -- posed no health threat, the Tokyo Metropolitan Government said.