The ht://Dig system is a complete world wide web indexing and searching system for a small domain or intranet. This system is not meant to replace the need for powerful internet-wide search systems like Lycos, Infoseek, Webcrawler and AltaVista. Instead it is meant to cover the search needs for a single company, campus, or even a particular sub section of a web site.
As opposed to some WAIS-based or web-server based search engines, ht://Dig can span several web servers at a site. The type of these different web servers doesn't matter as long as they understand the HTTP 1.0 protocol.
Features:
* Intranet searching * It is free * Robot exclusion is supported * Boolean expression searching * Configurable search results * Fuzzy searching * Searching of HTML and text files * Keywords can be added to HTML documents * Email notification of expired documents * A Protected server can be indexed * Searches on subsections of the database * Full source code included * The depth of the search can be limited * Full support for the ISO-Latin-1 character set
Disk space requirements:
The search engine will require lots of disk space to store its databases. Unfortunately, there is no exact formula to compute the space requirements. It depends on the number of documents you are going to index but also on the various options you use. To give you an idea of the space requirements, here is what I have deduced from our own database size at San Diego State University.
If you keep around the wordlist database (for update digging instead of initial digging) I found that multiplying the number of documents covered by 12,000 will come pretty close to the space required.
We have about 13,000 documents: 150MB index size with a 'wordlist' database
93MB index size without a 'wordlist' database
The package is available in two varieties, the 'stable', well-tested version (this one) and a less tested version (as 'htdig3.2').
|
|
|
| Architecture | Package Size | Installed Size | Files |
|---|---|---|---|
| alpha | 1,111.7 kB | 3820 kB | [list of files] |
| amd64 (unofficial port) | 986.7 kB | 3108 kB | [list of files] |
| arm | 991.5 kB | 3276 kB | [list of files] |
| hppa | 1,110.5 kB | 3900 kB | [list of files] |
| i386 | 944.5 kB | 2880 kB | [list of files] |
| ia64 | 1,285.8 kB | 5740 kB | [list of files] |
| m68k | 939.5 kB | 2896 kB | [list of files] |
| mips | 1,039.5 kB | 4500 kB | [list of files] |
| mipsel | 1,030.6 kB | 4496 kB | [list of files] |
| powerpc | 968.9 kB | 3296 kB | [list of files] |
| s390 | 950.1 kB | 3324 kB | [list of files] |
| sparc | 964.9 kB | 3448 kB | [list of files] |