macos Darwin安装faiss-cpu

文章目录

macos 使用brew instll fass, 后python3.12执行引用faiss包功能出现的问题

安装时遇到问题如下

复制代码

ModuleNotFoundError                       Traceback (most recent call last)
File ~/Src/ai/framework/langchain/.venv/lib/python3.12/site-packages/langchain_community/vectorstores/faiss.py:55, in dependable_faiss_import(no_avx2)
     54     else:
---> 55         import faiss
     56 except ImportError:

ModuleNotFoundError: No module named 'faiss'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
Cell In[1], line 19
     14 texts = text_splitter.create_documents(
     15     [page.page_content for page in pages[:4]]
     16 )
     18 embeddings = OpenAIEmbeddings(model="text-embedding-ada-002")
---> 19 db = FAISS.from_documents(texts, embeddings)
     21 retriever = db.as_retriever(search_kwargs={"k": 5})
     23 docs = retriever.invoke(query="llama2有多少参数?")

File ~/Src/ai/framework/langchain/.venv/lib/python3.12/site-packages/langchain_core/vectorstores.py:635, in VectorStore.from_documents(cls, documents, embedding, **kwargs)
    633 texts = [d.page_content for d in documents]
    634 metadatas = [d.metadata for d in documents]
--> 635 return cls.from_texts(texts, embedding, metadatas=metadatas, **kwargs)

File ~/Src/ai/framework/langchain/.venv/lib/python3.12/site-packages/langchain_community/vectorstores/faiss.py:931, in FAISS.from_texts(cls, texts, embedding, metadatas, ids, **kwargs)
    912 """Construct FAISS wrapper from raw documents.
    913 
    914 This is a user friendly interface that:
   (...)
    928         faiss = FAISS.from_texts(texts, embeddings)
    929 """
    930 embeddings = embedding.embed_documents(texts)
--> 931 return cls.__from(
    932     texts,
    933     embeddings,
    934     embedding,
    935     metadatas=metadatas,
    936     ids=ids,
    937     **kwargs,
    938 )

File ~/Src/ai/framework/langchain/.venv/lib/python3.12/site-packages/langchain_community/vectorstores/faiss.py:883, in FAISS.__from(cls, texts, embeddings, embedding, metadatas, ids, normalize_L2, distance_strategy, **kwargs)
    871 @classmethod
    872 def __from(
    873     cls,
   (...)
    881     **kwargs: Any,
    882 ) -> FAISS:
--> 883     faiss = dependable_faiss_import()
    884     if distance_strategy == DistanceStrategy.MAX_INNER_PRODUCT:
    885         index = faiss.IndexFlatIP(len(embeddings[0]))

File ~/Src/ai/framework/langchain/.venv/lib/python3.12/site-packages/langchain_community/vectorstores/faiss.py:57, in dependable_faiss_import(no_avx2)
     55         import faiss
     56 except ImportError:
---> 57     raise ImportError(
     58         "Could not import faiss python package. "
     59         "Please install it with `pip install faiss-gpu` (for CUDA supported GPU) "
     60         "or `pip install faiss-cpu` (depending on Python version)."
     61     )
     62 return faiss

ImportError: Could not import faiss python package. Please install it with `pip install faiss-gpu` (for CUDA supported GPU) or `pip install faiss-cpu` (depending on Python version).

直接使用本地 pip安装fass-cpu会遇到如下问题

复制代码

venwei@svenweis-MacBook-Pro langchain$ pip3 install faiss-cpu 
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a Python library that isn't in Homebrew,
    use a virtual environment:
    
    python3 -m venv path/to/venv
    source path/to/venv/bin/activate
    python3 -m pip install xyz
    
    If you wish to install a Python application that isn't in Homebrew,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. You can install pipx with
    
    brew install pipx
    
    You may restore the old behavior of pip by passing
    the '--break-system-packages' flag to pip, or by adding
    'break-system-packages = true' to your pip.conf file. The latter
    will permanently disable this error.
    
    If you disable this error, we STRONGLY recommend that you additionally
    pass the '--user' flag to pip, or set 'user = true' in your pip.conf
    file. Failure to do this can result in a broken Homebrew installation.
    
    Read more about this behavior here: <https://peps.python.org/pep-0668/>
    ```
    大概意思是让尝试使用brew来安装, 安装完,再使用python来执行文件就出现文章头的错误. 提示内容也说如果非要使用pip安装就使用python虚拟环境. 目前使用python版本为python3.12, 于是就创建了虚拟环境,并且再次使用pip install faiss-cpu, 又遇到如下列表问题,
* 缺少其它安装包
* swig 执行时出现不同的错误
* ...
中间考虑过是否为python版本问题,运行看pypi上看下说明,只是大于Requires: Python >=3.8, 这句直接导致我浪费大量时间,中间也切换过python3.11遇到是swig执行时遇到问题.
于是考虑是不是swig版本与faiss的版本有相关的,又尝试不同的swig版本,结果仍出问题.
最后想编译faiss,在打算着手做时,才发现python的版本要求是3.8~3.10..WTF
下载并使用python3.10安装faiss-cpu,没用几分钟成功安装