webmagic 爬取https的网站抛avax.net.ssl.SSLHandshakeException异常

IT修炼之路2024-04-23 21:14

webmagic 抓取带有https的网站，抛出的异常javax.net.ssl.SSLHandshakeException。

初步解决办法：

1,在自己的项目中新建httpclient文件夹，新建类HttpClientGenerator, 复制webmagic源码中的 HttpClientGenerator.

2.修改 HttpClientGenerator 的代码，需要修改 buildSSLConnectionSocketFactory 这个方法。

java 复制代码

private SSLConnectionSocketFactory buildSSLConnectionSocketFactory() {
		try {
            return new SSLConnectionSocketFactory(createIgnoreVerifySSL(), new String[]{"SSLv2Hello","SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2"},
                    null,
                    new DefaultHostnameVerifier()); // 优先绕过安全证书
		} catch (KeyManagementException e) {
            logger.error("ssl connection fail", e);
        } catch (NoSuchAlgorithmException e) {
            logger.error("ssl connection fail", e);
        }
		return SSLConnectionSocketFactory.getSocketFactory();
	}

3,修改 HttpClientDownloader 中引用的 HttpClientGenerator 为你修改后的类。

4.设置爬虫 Spider 的 Downloader 为你修改的 HttpClientDownloader。

做以上修改之后如果问题依然没解决，报错：

SSLException: Certificate for *** doesn't match any of the subject alternative

此错误是说明校验证书和域名失败，绕过就可以了。

正常情况下SSL连接会验证码所有证书信息

.register("https", new SSLConnectionSocketFactory(sslcontext)).build();

修改HttpClientGenerator的构造方法跳过验证，注释掉的代码为源码：

java 复制代码

 public HttpClientGenerator() {
//        Registry<ConnectionSocketFactory> reg = RegistryBuilder.<ConnectionSocketFactory>create()
//                .register("http", PlainConnectionSocketFactory.INSTANCE)
//                .register("https", buildSSLConnectionSocketFactory())
//                .build();
//        SSLContext sslcontext = sslContext(keyStorePath, keyStorePassword);
        SSLContext sslcontext = null;
        try {
            sslcontext = createIgnoreVerifySSL();
        } catch (NoSuchAlgorithmException e) {
            throw new RuntimeException(e);
        } catch (KeyManagementException e) {
            throw new RuntimeException(e);
        }
        Registry<ConnectionSocketFactory> reg = RegistryBuilder.<ConnectionSocketFactory>create()
                .register("http", PlainConnectionSocketFactory.INSTANCE)
                //  只忽略域名验证码
                .register("https", new SSLConnectionSocketFactory(sslcontext, NoopHostnameVerifier.INSTANCE)).build();

        connectionManager = new PoolingHttpClientConnectionManager(reg);
        connectionManager.setDefaultMaxPerRoute(100);
    }