The .web file format is a JSON file with a .web extension used to configure website crawling for RAG Assistant.
The following parameters are available in the .web file format:
| Element Name |
DataType |
Description |
| url |
string |
Mandatory: the base URL to crawl. |
| excludeDirs |
string[] |
Optional list of web page directories to exclude; they must be absolute references.
Here you can exclude unnecessary dependencies referenced in the HTML. |
| maxDepth |
number |
The maximum depth to crawl. By default, it is set to 3; the system controls it cannot have a higher value. |
| timeout |
number |
Timeout for each request, in the unit of milliseconds. By default, it is set to 10000 (10 seconds). |
| preventOutside |
boolean |
Whether to prevent crawling outside the root base URL; true by default. |
| selectors |
string[] |
List of page selectors to consider; it uses the body tag by default, and can be used in the following combinations:
*
- universal selector;div
- tag name;.foo
- class name;#bar
- id;[baz]
- attribute presence;[baz=buzz]
- attribute value (with any operators and also quotes and case sensitivity modifiers - syntax;+
and >
combinators (other combinators are not supported).
It is recommended to analyze some of the web pages of the site and select the main container where the valuable text is set.
|
| linkBrackets |
boolean |
Add link brackets when processing an anchor tag; false by default. |
| ignoreHref |
boolean |
Remove href references; false by default. |
Once the file is uploaded (as shown in Upload documents), the ingestion process will crawl the site starting in the base URL (url parameter) considering the maxDepth parameter (to follow associated links in each page up-to maxDepth) and will exclude all base URLs (specified in the excludeDirs parameter).
When the process is finished, the crawl status will be displayed as usual, in this case associated to the uploaded file (in general as Success):
You can check the Chunks file with the result of the processing, and validate that the detected plain text by the crawler makes sense (to be used as semantic information for a RAG assistant).
Suppose you want to crawl a Wikipedia article. In this case, a possible wikipedia_sample.web file to upload could be configured as follows:
{
"url": "https://en.wikipedia.org/wiki/GeneXus",
"maxDepth": 1,
"preventOutside": true,
"timeout": 10000,
"selectors": [
"#firstHeading",
"#bodyContent"
],
"linkBrackets": false,
"ignoreHref": true,
"excludeDirs": [
"https://en.wikipedia.org/static/",
"https://en.wikipedia.org/w/"
]
}
Note the following:
- url: The base URL of the Wikipedia article.
- maxDepth: Only crawl one level of the site.
- selectors: Use the selectors by ID (firstHeading, bodyContent) to only process the important content from the site.
- linkBrackets: Remove link references to keep the plain text as clean as possible.
- excludeDirs: Exclude a couple of directories.
The resulting Chunks file will look something like this:
[
...
{
"pageContent": "GENEXUS From Wikipedia, the free encyclopedia Computer programming tool\nThis article is about a computer programming tool. For the Fear Factory album, see Genexus (album). GeneXusDeveloper(s)GeneXusInitial release1988; 36 years ago (1988)Stable release\nGeneXus 17\n...",
"metadata": {
"source": "https://en.wikipedia.org/wiki/GeneXus",
"language": "en",
"description": "GeneXus - Wikipedia",
"id": "someGUID",
}
},
{
"pageContent": "...can be generated include COBOL, Java, Objective-C, RPG, Ruby, Visual Basic, and Visual FoxPro.[3] Some of the DBMSs supported are\nMicrosoft SQL Server, Oracle, IBM Db2, Informix, PostgreSQL, and MySQL. GeneXus was developed by Uruguayan company ARTech Consultores SRL which later renamed to Genexus SA.[4] The latest version is...",
"metadata": {
"source": "https://en.wikipedia.org/wiki/GeneXus",
"language": "en",
"description": "GeneXus - Wikipedia",
"id": "someGUID",
}
}
...
]
.custom File Format