The rejection starts an automatic 15-day extension of the previous contract. The union will most likely continue to negotiate, said Barry Dixon, president of the International Union of Operating Engineers Local 953.
Miners turned back BHP Billiton's contract offer by a tally of 343 to 47. Eighty-six percent of union members cast ballots.
The union and company will soon discuss when to meet next, Dixon said.
The union did not recommend that members approve the contract.
"We had a brief discussion on their proposal, and we told the company we couldn't recommend the proposal to our membership," Dixon said.
Neither side would detail BHP Billiton's contract offer, or explain why it fell short for the miners.
"The company has provided a fair and reasonable contract offer considering the current economic environment and pressures on the business," said Norman Benally, a spokesman for BHP Billiton's New Mexico Coal unit, in a prepared statement.
The contract under negotiation covers surface miners at the two coal mines west of Farmington. Underground miners at San Juan Mine are covered by a separate labor agreement.
BHP Billiton operates both mines and is San Juan County's largest private employer with more than $100 million in payroll.
Navajo Mine is under pressure from a scheduled partial shutdown of Four Corners Power Plant, the mine's sole customer. The decommissioning of three of the plant's five stacks would reduce demand for the mine's coal by about 30 percent.
The impending slowdown in demand played a role in BHP's offer, Benally said.
"The offer provided to the union maintains a highly competitive compensation package with some changes necessary to ensure that the business is successful with reduced coal production in the future," he said.
Benally said the miners are among the highest paid and have some of the most generous benefits in the western U.S.
Their previous contract was set to expire at midnight Thursday. BHP and the union began negotiating a new contract in October, and the mining company tendered a "last, best and final offer" last Friday.
In December, BHP Billiton announced it had reached a preliminary agreement to sell Navajo Mine to the Navajo Nation. Under the memorandum of understanding, BHP would continue operating the mine for the tribe until July 2016
At that point, the tribe would take over with its own company, or another company of the tribe's choosing. Mine employees would stay on with the new operator.
"We do hope that they do come to an agreement," said Erny Zah, spokesman for Navajo Nation President Ben Shelly. "Having a contract in place would just further clarify the needed information that all of our leaders are seeking, from the council delegates to the president. We want to see hard numbers.
Pat Risner, president of BHP Billiton New Mexico Coal, told county commissioners in early January the company would cut about 100 jobs at the mine before the mine's midyear transfer to the tribe.
Risner said BHP would try to avoid layoffs, instead offering buyouts to many of the mine's employees who are past retirement age.