Aapche Nutch建立自己的搜索引擎

杨江2024-03-29 12:12

sudo apt install default-jdk'

java -version

openjdk version "11.0.22" 2024-01-16

vi .bashrc

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

mkdir -p urls

cd urls

touch seed.txt

里面放入我的网站地址

bin/nutch inject crawl/crawldb urls

显示

Injecting seed URL file file:/data/apache-nutch-1.19/urls/seed.txt

Total new urls injected: 1

s1=`ls -d crawl/segments/2* | tail -1`

echo $s1

bin/nutch generate crawl/crawldb crawl/segments

apache nutch No agents listed in 'http.agent.name' property.

conf/ nutch-site.xml

<name>http.agent.name</name>

<value>MyNutchBot/1.0</value>

</property>

export APACHE_SOLR_HOME=/data/solr-8.11.3

export NUTCH_RUNTIME_HOME=/data/apache-nutch-1.19

${APACHE_SOLR_HOME}/bin/solr start -force

open file limit is currently 1024

vi /etc/security/limits.conf

* soft nofile 4096

* hard nofile 4096

Started Solr server on port 8983 (pid=29369). Happy searching!

${APACHE_SOLR_HOME}/bin/solr start -force

${APACHE_SOLR_HOME}/bin/solr create -c nutch -d ${APACHE_SOLR_HOME}/server/solr/configsets/nutch/conf/ -force

ls crawl/segments/

bin/nutch index crawl/crawldb/ -linkdb crawl/linkdb/ crawl/segments/20240326063028/ -filter -normalize -deleteGone

去掉robot的处理