|
|
How can I add crawl-it search capability to my web site?
|
|
|
1. Register your internet address with crawl-it.
2. Add a search entry field to your web site. (The HTML code you need is provided by crawl-it.)
|
 |
|
|
What is a search entry field?
|
|
|
A search entry field is a text entry field on your web site. Visitors to your web site can enter any keywords into the search entry field. Usually it is placed on the home or start page.
|
|
|
|
How do I add the search entry field to my web pages?
|
|
|
You only have to add a few lines of HTML code to one of the pages on your web site. The HTML code creates the search entry field in which users can enter the keywords they are searching for. You can copy the required HTML code from your personal members' area at any time.
|
|
|
|
Which pages does crawl-it search? |
|
|
Crawl-it searches all pages contained "below" and including the URL you provided. Crawl-it does NOT index other internet sites you may have referenced. I.e. only pages belonging to your own site are indexed.
|
|
|
|
How can I archive that certain webpages will be displayed at the top of the search result list?
|
|
|
You can assign webpages to groups. Pages that have been assigned to groups will always be displayed at the top of the result list. To assign a page to a group you need to add the following HTML tag to your page:
<meta NAME="groupname" content="XXX"> where XXX is the group name that has been choosen by you. This group name will be displayed on top of your result list.
|
|
|
|
Does crawl-it support frames? |
|
|
Yes, crawl-it can handle frames. In fact, crawl-it remembers the frame in which an indexed page was contained and displays the complete frame and not only the single page. This way, your site layout remains the same when slecting a link from the hit list.
|
|
|
|
What does crawl-it do with pages containing JavaScript? |
|
|
Crawl-it knows how to index pages containing JavaScript. However, the URL's referenced in the JavaScript must be fully available on the following pages. Crawl-it cannot index pages which are created dynamically using JavaScript.
|
|
|
|
How does crawl-it know that pages have been modified? |
|
|
During indexing, the last modification date of your page is returned. Only pages which have been modified are re-indexed.
|
|
|
|
Does crawl-it recognize robot.txt? |
|
|
Yes. We recognize and respect the entries in your robot.txt file. Further information on the contents of this file can be found under http://www.robotstxt.org/wc/norobots.html
|
|
|
|
Can I prevent crawl-it from indexing a particular page? |
|
|
Yes! You can disable files as well as directories from indexing. One option to this is to write a robots.txt file, which needs to be placed in the root context of your web server domain. Within this file you define the directories and files which will be disabled from indexing. A detailed description on how to do this can be found at:
http://www.robotstxt.org/wc/norobots.html
Our crawler agent is been called "crawl-it".
You can also disable pages from being indexed by adding the following tag:
<meta name="robots" content="noindex,follow">
You can use the following options:
| INDEX | Index this page |
| NOINDEX | Do not index this page |
| FOLLOW | Index the pages that are been referred on this page |
| NOFOLLOW | Don't follow links on this page |
You can also disable parts of text on a page from being indexed through this tag:
<!-- crawl-it no index --> Text to not be indexed <!--crawl-it end no index-->
|
|
|
|
Can crawl-it search multiple servers at the same time? |
|
|
Yes. Crawl-it can provide search functionality over multiple URL's. Register using your main URL, then enter additional "entry points" via your members' area. Then you can administer different internet sites using one account.
|
|
|
|
Can I get search statistics from crawl-it?
|
|
|
Yes. Crawl-it offers a number of different statistics for analysis. Your password protected members' area contains a section with statistics on the search behaviour of visitors to your site. These are:
1. The number of search requests per day or over a number of days.
2. The most recently entered search keywords.
3. The most frequently entered search keywords entered by your users over a particular period.
4. Search keywords entered by your users which did not return any hits (pages).
By viewing the phrases your users are entering, you can easily modify your web site to provide the information they are looking for.
|
|
|
|
What does crawl-it cost? |
|
|
Crawl-it can be used for free. This is made possible by the including ads in the hit list. If you prefer the ad-free service, then you can select one of the paid services. Depending on your requirements (size of your web site, indexing frequency) we have a service package to suit you.
|
|
|
|
What are the crawl-it terms of notice? |
|
|
You can give notice for the Free&Fun tariff at any time. The paid services are each paid for a period of three months, and if no notice is given, are extended for a further three months. On request, notice can be given up to one week before the end of the three month period.
|
|
|
|
How do I get the short description of my pages to display a concise text? |
|
|
The short description displayed for each page on the hit list usually consists of the first 200 characters found on the page. However, you can create an alternate text to be displayed. To do so you can add the following HTML text to the page header.
<META NAME="DESCRIPTION" CONTENT="This is the short description.">
|
|
|
|
What does "indexing" mean? |
|
|
Indexing refers to the process whereby your website is searched for relevant information. Crawl-it automatically searches your website at regular intervals and creates a list of contents of your site. Each time a user enters a keyword in the search entry field and begins a search of your site, the list of contents is checked and compared with the search keywords.
|
|
|
|
Does crawl-it support special characters like German umlauts? |
|
|
Yes! Crawl-it has been optimized for the German language.
|
|
|
|
Can I modify the layout of the hit list to suit my requirements? |
|
|
Yes. Crawl-it allows unlimited freedom in designing the layout of the hit list. You can either select one of the predefined layouts, you can set the background color, link color, font and font size, or you can use the Expert Mode to define the layout completely independently, and provide crawl-it with the source code which is to be included when diusplaying the hit list. All required steps can be done easily via your personal members' area.
|
|
|
|
Is there a way to define which subset of URLs will be displayed on the result page?
|
|
|
Yes - through the definition of a substring you can define which Urls will be displayed on your result page. Add the following tag behind the <FORM> tag of your search field:
<input type="hidden" name="p_substring" value="XXX"/>
Replace XXX with the substring that needs to be in the URL of any result page. Example:
<input type="hidden" name="p_substring" value="/news">
This will only show result pages that have the substring "/news" included in the result URL (e.g. www.xyz/news/index.html). If you want to exclude a substring from the result urls use this:
<input type="hidden" name="p_substring" value="-news">
You would see all pages where the url DOES NOT contain the substring "news".
|
|
|
|
How can I get technical support? |
|
|
Crawl-it customers get technical support via email. We will attempt to solve all technical problems which may occur as quickly as possible.
|
|
|
|
Can crawl-it index my intranet? |
|
|
No. Crawl-it cannot cross your firewall.
|
|
|
|
Can crawl-it index password-protected pages? |
|
|
Yes. Simply add the username and password into the appropriate fields in your members area under the section "user profile".
|
|
|
|
What is a "What's New" list? |
|
|
The "What's New" list is a list which crawl-it automatically creates of links on your web site which have been created or modified within the last 4 weeks. This list is particularly useful to users of your site who wish to see which areas have been updated.
|
|
|
|
How do I create a "What's New" list? |
|
|
This page is created automatically by crawl-it. In your members' area, under "HTML Code", you will find sample HTML code which creates a link. All you need to do is add this link to your web page.
|
|
|
|
Which kinds of search strings can users enter?
|
|
|
The user can enter one or more words in the search entry field. If multiple words are entered, all pages are listed which contain at least one of the entered words. By using the "+" symbol, for "AND", the user can search for pages containing both or all words entered. Or by using the "-" symbol, pages can be found which do NOT contain the entered word. For example:
a search for: +food -drink
will return all pages which contain the word "food" but not the word "drink".
|
|
|
|
I use frames on my pages. Can I integrate the search entry field in such a way that the result list is displayed in a different frame? |
|
|
Yes. To do so you have to modify the HTML code provided by crawl-it slightly. Replace
<FORM TARGET=....
with
<FORM TARGET="remoteframe" ...
where "remoteframe" is the name of the frame in which the result list is to be displayed.
|
|
|
|
Can I replace the "Search" button next to the search entry field with an image? |
|
|
Yes. To do so you have to modify the HTML code provided by crawl-it slightly. Replace:
<INPUT TYPE=SUBMIT...
with
<input type="IMAGE" src="image.gif" border ="0" name="search">
and replace "image.gif" with the name of the required image.
|
|
|
|
I have a Java menu. Will crawl-it find the URL's listed there? |
|
|
Usually not! However, there is a way to work around this. Re-enter the URL's in your HTML source code, then comment them out. This way, the URL's are not visible to users of the web page, but the crawler can still index the pages.
|
|
|
|
I don't see any entries in my site map. Why not? |
|
|
Crawl-it only displays the site map for the URL which was entered when you registered. If you entered additional URL entry points and the pages are only located under these additional URL's then the site map entries won't be displayed.
|
|
|
|
My PDF documents do not get indexed. Why? |
|
|
PDF indexing has to be purchased seperately. PDF indexing is available at 5$ per month or 50$ per year.
|
|
|
|
I cannot log in, even though my Username is correct... |
|
|
The following reasons could be valid:
- You have disabled your cookie settings in your browsers. Crawl-it requires that the cookie option is enabled.
- The server is overloaded. Please re-try at a later stage.
|
|
|
|
Is crawl-it able to index pages that are generated through a database? |
|
|
Crawl-it can index all HTML pages that are accessable through a browser and that are linked together. The content of these HTML can be generated through a database.
|
|
|
|
How can I disable my crawl-it account? |
|
|
Use the support option to inform us, that you wish to stop using your crawl-it account. We will disable your account immediately.
|
|
