Houghton Mifflin Co. on Nov. 29 agreed to a nearly $1.8 billion buyout that will pair a venerable name in educational textbook publishing with Riverdeep Inc., a smaller firm whose complementary strength is in educational software.

By joining Houghton Mifflin's paper-and-ink business with Riverdeep's core strength in educational software, the newly formed venture hopes to better position itself against larger rivals Pearson PLC, McGraw-Hill Cos., and Harcourt Education.

The deal is the second significant merger in just over a month that is sure to affect the educational software market. In late October, Adobe Systems purchased Serious Magic, a maker of video editing tools, for an undisclosed amount (see related story).

Barry O'Callaghan, Riverdeep's chairman and CEO, said the transaction seeks to "capitalize on the convergence of print and digital education platforms" and help Riverdeep draw strength from Houghton Mifflin's larger sales force.

The buyers, a newly formed holding company called HM Rivergroup PLC, are led by O'Callaghan, a former investment banker from Ireland and a controlling shareholder of Riverdeep.

HM Rivergroup is paying $1.75 billion in cash for Boston-based Houghton Mifflin. In addition, some Houghton managers and employees are rolling over $40 million in stock into shares of the new company.

The buyers also are assuming about $1.61 billion in debt in the deal, expected to create a company with more than $1.4 billion in revenue--most of it from Houghton Mifflin, which has about 3,500 employees, including 1,200 in Boston, to Riverdeep's total 300 workers.

As part of the deal, the new holding group also acquired Riverdeep in a share-for-share exchange valuing Riverdeep at approximately $1.2 billion, including assumption of debt, bringing the total purchase to more than $5 billion.

After the deal's expected completion by year's end, the new holding company will be based in Ireland and be renamed Houghton Mifflin Riverdeep Group PLC.

Houghton Mifflin, the fourth-largest textbook publisher in the United States, traces its origins to 1832. It was home to early American authors such as Ralph Waldo Emerson and Henry David Thoreau, as well as British author J.R.R. Tolkien.

The Nov. 29 transaction was the latest in a string of ownership changes for Houghton Mifflin, which was acquired by French media and telecommunications company Vivendi Universal in 2001 for $2.2 billion, including $500 million in assumed debt. The next year, Vivendi sold Houghton Mifflin for $1.28 billion to Thomas H. Lee and Bain Capital, two Boston firms that also agreed to assume $380 million of debt.

Riverdeep was founded in 1995 and underwent a management buyout led by O'Callaghan in 2002. The company's web-based and CD-ROM course materials include its flagship product, called Destination Success. The firm has recently added education products from such names as The Learning Company, Broderbund, and Edmark.