The Hidden Markov Model Toolkit (HTK) is a portable toolkit for building and manipulating hidden Markov models. HTK is primarily used for speech recognition research although it has been used for numerous other applications including research into speech synthesis, character recognition and DNA sequencing. HTK is in use at hundreds of sites worldwide.
Quoted from the HTK website.

HTK is a free/open-source piece of software. It builds on Linux, Solaris, IRIX, HPUX, Mac OS/X, Windows NT, 2000, XP and FreeBSD.

The problem:
If you tried to build HTK on a command-line-only Linux machine (no X installed), it will give build errors. You may face this situation if you need to run your experiments on a hosted solution that has no GUI to exploit its CPU/memory power. The problem happens because there is a single tool that requires graphical user interface: HSLab.

HSLab is an interactive label editor for manipulating speech label files. An example of using HSLab would be to load a sampled waveform file, determine the boundaries of the speech units of interest and assign labels to them. HSLab is the only tool in the HTK package which makes use of the graphics library HGraf.
In most cases you will not need this tool in your regular experiments. So if this is your case, you can disable it so that the rest of HTK would build successfully.

The solution:
I assume you are using the latest version of HTK (3.4). Disabling HSLab comes in 2 steps:
  1. Remove HSLab target from file HTKTools/Makefile in line 48.
  2. Remove HGraph from object dependancy in file HTKLib/Makefile lines 49 and 78.
If you have no idea how to make this, you can download my 2 patches and place them in HTK src directory. Run the following commands to patch the makefiles and make/install HTK:

# change directory to htk src path, then:
patch HTKLib/Makefile HTKLib.Makefile.patch
patch HTKTools/Makefile HTKTools.Makefile.patch
sudo make install

Now the whole HTK installs with no problems.
Enjoy :)