diff --git a/sources/platform/integrations/ai/haystack.md b/sources/platform/integrations/ai/haystack.md index f8b39da02c..4b653f137a 100644 --- a/sources/platform/integrations/ai/haystack.md +++ b/sources/platform/integrations/ai/haystack.md @@ -1,187 +1,235 @@ --- title: Haystack integration sidebar_label: Haystack -description: Learn how to integrate Apify with Haystack to crawl websites, compute text embeddings, and build retrieval-augmented generation pipelines with ease. +description: Integrate Apify with Haystack to load web content into RAG pipelines, search the web, and give Haystack agents access to any Actor from Apify Store. slug: /integrations/haystack --- import ThirdPartyDisclaimer from '@site/sources/_partials/_third-party-integration.mdx'; -[Haystack](https://haystack.deepset.ai/) is an open source framework for building production-ready LLM applications, agents, advanced retrieval-augmented generative pipelines, and state-of-the-art search systems that work intelligently over large document collections. For more information on Haystack, visit its [documentation](https://docs.haystack.deepset.ai/docs/intro). +[Haystack](https://haystack.deepset.ai/) is an open-source framework by deepset for building LLM applications, retrieval-augmented generation (RAG) pipelines, and AI agents. To learn more, see the [Haystack documentation](https://docs.haystack.deepset.ai/docs/intro). -In this example, we'll use the [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor, which can deeply crawl websites such as documentation sites, knowledge bases, or blogs, and extract text content from the web pages. -Then, we'll use the `OpenAIDocumentEmbedder` to compute text embeddings and the `InMemoryDocumentStore` to store documents in a temporary in-memory database. -The last step will be to retrieve the most similar documents. +The [`apify-haystack`](https://pypi.org/project/apify-haystack/) package connects Haystack to the [Apify platform](https://apify.com/). It provides these components: -This example uses the Apify-Haystack Python integration published on [PyPi](https://pypi.org/project/apify-haystack/). -Before we start with the integration, we need to install all dependencies: +- `ApifyDatasetFromActorCall` runs an Actor, waits for it to finish, and loads its default dataset as Haystack `Document` objects. +- `ApifyDatasetFromTaskCall` does the same for a saved [Actor task](/actors/running/tasks). +- `ApifyDatasetLoader` loads an existing [dataset](/storage/dataset) by its ID. +- `ApifyWebSearch` searches the web with the [RAG Web Browser](https://apify.com/apify/rag-web-browser) Actor and returns the content of the top result pages. +- `ApifyActorTool` lets a Haystack `Agent` run any Actor and work with its results. -```bash -pip install apify-haystack haystack-ai -``` + -Import all required packages: +## Prerequisites -```python -from haystack import Document, Pipeline -from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder -from haystack.components.preprocessors import DocumentCleaner, DocumentSplitter -from haystack.components.retrievers import InMemoryBM25Retriever, InMemoryEmbeddingRetriever -from haystack.components.writers import DocumentWriter -from haystack.document_stores.in_memory import InMemoryDocumentStore -from haystack.utils.auth import Secret +- Python 3.11 or newer +- An [Apify account](https://console.apify.com/sign-up) and your API token from the [Integrations](https://console.apify.com/settings/integrations) page in Apify Console +- An [OpenAI API key](https://platform.openai.com/account/api-keys) for the examples on this page -from apify_haystack import ApifyDatasetFromActorCall -``` +Haystack supports many other model providers too. -Find your [Apify API token](https://console.apify.com/settings/integrations) and [OpenAI API key](https://platform.openai.com/account/api-keys) and initialize these into environment variable: +## Installation -```python -import os +Install Haystack and the Apify integration: -os.environ["APIFY_API_TOKEN"] = "YOUR-APIFY-API-TOKEN" -os.environ["OPENAI_API_KEY"] = "YOUR-OPENAI-API-KEY" +```bash +pip install haystack-ai apify-haystack ``` -First, you need to create a document loader that will crawl the haystack website using the Website Content Crawler: +Set your Apify API token and OpenAI API key as environment variables: -```python -document_loader = ApifyDatasetFromActorCall( - actor_id="apify/website-content-crawler", - run_input={ - "maxCrawlPages": 3, # limit the number of pages to crawl - "startUrls": [{"url": "https://haystack.deepset.ai/"}], - }, - dataset_mapping_function=lambda item: Document(content=item["text"] or "", meta={"url": item["url"]}), -) +```bash +export APIFY_TOKEN="YOUR-APIFY-API-TOKEN" +export OPENAI_API_KEY="YOUR-OPENAI-API-KEY" ``` -You can learn more about input parameters on the [Website Content Crawler inputs page](https://apify.com/apify/website-content-crawler/input-schema). -The dataset mapping function is described in more detail in the [Retrieval augmented generation example](https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/apify_haystack_rag.ipynb). +The components read the token from `APIFY_TOKEN`, or from `APIFY_API_TOKEN` if `APIFY_TOKEN` isn't set. To read it from another variable, pass a Haystack `Secret`, for example `apify_token=Secret.from_env_var("MY_APIFY_TOKEN")`. -Next, you can utilize the [Haystack pipeline](https://docs.haystack.deepset.ai/docs/pipelines), which helps you connect several processing components together. I -n this example, we connect the document loader with the document splitter, document embedder, and document writer components. +## Answer questions about website content -```python -document_store = InMemoryDocumentStore() +This example crawls a part of the Haystack documentation with [Website Content Crawler](https://apify.com/apify/website-content-crawler), stores the pages in a document store, and answers questions about them. + +1. Crawl the pages, split them into chunks, compute their embeddings, and store them: + + ```python + from haystack import Pipeline + from haystack.components.embedders import OpenAIDocumentEmbedder + from haystack.components.preprocessors import DocumentSplitter + from haystack.components.writers import DocumentWriter + from haystack.document_stores.in_memory import InMemoryDocumentStore + + from apify_haystack import ApifyDatasetFromActorCall + + document_store = InMemoryDocumentStore() + + crawler = ApifyDatasetFromActorCall( + actor_id="apify/website-content-crawler", + run_input={ + "startUrls": [{"url": "https://docs.haystack.deepset.ai/docs/intro"}], + "maxCrawlPages": 10, + }, + ) + + indexing = Pipeline() + indexing.add_component("crawler", crawler) + indexing.add_component("splitter", DocumentSplitter(split_by="word", split_length=150, split_overlap=50)) + indexing.add_component("embedder", OpenAIDocumentEmbedder()) + indexing.add_component("writer", DocumentWriter(document_store)) + indexing.connect("crawler", "splitter") + indexing.connect("splitter", "embedder") + indexing.connect("embedder", "writer") + + indexing.run({}) + print(f"Stored {document_store.count_documents()} documents") + ``` + + The crawler turns each page into a `Document` with the page Markdown as its content and the page `url`, `title`, and `description` in `meta`. To change what it crawls, see the [Website Content Crawler input](https://apify.com/apify/website-content-crawler/input-schema). The crawl can take a few minutes. You can follow it on the [Runs](https://console.apify.com/actors/runs) page in Apify Console. + +1. Retrieve the chunks closest to a question and let the LLM answer it: + + ```python + from haystack.components.builders import ChatPromptBuilder + from haystack.components.embedders import OpenAITextEmbedder + from haystack.components.generators.chat import OpenAIChatGenerator + from haystack.components.retrievers import InMemoryEmbeddingRetriever + from haystack.dataclasses import ChatMessage + + template = [ + ChatMessage.from_user( + """Answer the question based on the documents. + + Documents: + {% for document in documents %} + {{ document.content }} + {% endfor %} + + Question: {{ question }}""" + ) + ] -document_splitter = DocumentSplitter(split_by="word", split_length=150, split_overlap=50) -document_embedder = OpenAIDocumentEmbedder() -document_writer = DocumentWriter(document_store) + querying = Pipeline() + querying.add_component("embedder", OpenAITextEmbedder()) + querying.add_component("retriever", InMemoryEmbeddingRetriever(document_store, top_k=3)) + querying.add_component("prompt_builder", ChatPromptBuilder(template=template, required_variables="*")) + querying.add_component("llm", OpenAIChatGenerator()) + querying.connect("embedder.embedding", "retriever.query_embedding") + querying.connect("retriever", "prompt_builder.documents") + querying.connect("prompt_builder", "llm") -pipe = Pipeline() -pipe.add_component("document_loader", document_loader) -pipe.add_component("document_splitter", document_splitter) -pipe.add_component("document_embedder", document_embedder) -pipe.add_component("document_writer", document_writer) + question = "What is a Haystack pipeline?" + result = querying.run({"embedder": {"text": question}, "prompt_builder": {"question": question}}) + print(result["llm"]["replies"][0].text) + ``` -pipe.connect("document_loader", "document_splitter") -pipe.connect("document_splitter", "document_embedder") -pipe.connect("document_embedder", "document_writer") +## Search the web + +`ApifyWebSearch` runs the RAG Web Browser Actor, which searches Google, scrapes the top results, and returns their content as Markdown. It has the same outputs as other Haystack web search components, so you can use it as a drop-in replacement in a pipeline: + +```python +from apify_haystack import ApifyWebSearch + +search = ApifyWebSearch(top_k=3) +result = search.run(query="What is new in Haystack 3?") + +for document in result["documents"]: + print(document.meta["url"], document.meta.get("title")) ``` -Run all the components in the pipeline: +The `documents` output holds the result pages, and `links` holds their URLs. To pass more input to the Actor, use `search_params`, for example `search_params={"scrapingTool": "browser-playwright"}` for pages that need a browser to render. + +## Use Actors as agent tools + +`ApifyActorTool` turns any Actor into a tool for a Haystack `Agent`. The tool takes its parameters from the Actor's input schema and its description from the Actor's description, so the LLM knows how to call it: ```python -pipe.run({}) +from haystack.components.agents import Agent +from haystack.components.generators.chat import OpenAIChatGenerator +from haystack.dataclasses import ChatMessage + +from apify_haystack import ApifyActorTool + +browser = ApifyActorTool("apify/rag-web-browser", dataset_items_limit=3, max_total_charge_usd=0.5) +agent = Agent(chat_generator=OpenAIChatGenerator(), tools=[browser]) + +result = agent.run([ChatMessage.from_user("What is new in Haystack 3?")]) +print(result["last_message"].text) ``` -:::note Crawling may take some time +Note that: -The Actor call may take some time as it crawls the Haystack website. +- `dataset_items_limit` caps the number of dataset items the tool returns, which keeps its output within the model's context window. +- `max_total_charge_usd` caps the cost of each run the agent starts with a [pay-per-event](/actors/running/actors-in-store#pay-per-event) Actor. +- The tool wraps the items it returns in markers that tell the LLM to treat them as untrusted external content, which helps guard against prompt injection in scraped pages. -::: +## Load data from tasks and datasets -After running the pipeline code, you can print the results +To run a saved Actor task, use `ApifyDatasetFromTaskCall` with the task ID. The task runs with its saved input. To change some of its fields for a run, pass them in the optional `task_input`: ```python -print(f"Added {document_store.count_documents()} to vector from Website Content Crawler") +from apify_haystack import ApifyDatasetFromTaskCall -print("Retrieving documents from the document store using BM25") -print("query='Haystack'") -bm25_retriever = InMemoryBM25Retriever(document_store) -for doc in bm25_retriever.run("Haystack", top_k=1)["documents"]: - print(doc.content) +task = ApifyDatasetFromTaskCall(task_id="YOUR-TASK-ID", task_input={"maxCrawlPages": 5}) +documents = task.run()["documents"] ``` -If you want to test the whole example, you can simply create a new file, `apify_integration.py`, and copy the whole code into it. +To load a dataset that already exists, for example the result of an earlier run, use `ApifyDatasetLoader`: ```python -import os +from apify_haystack import ApifyDatasetLoader -from haystack import Document, Pipeline -from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder -from haystack.components.preprocessors import DocumentSplitter -from haystack.components.retrievers import InMemoryBM25Retriever, InMemoryEmbeddingRetriever -from haystack.components.writers import DocumentWriter -from haystack.document_stores.in_memory import InMemoryDocumentStore +loader = ApifyDatasetLoader(dataset_id="YOUR-DATASET-ID") +documents = loader.run()["documents"] +``` -from apify_haystack import ApifyDatasetFromActorCall +## Map dataset items to documents -os.environ["APIFY_API_TOKEN"] = "YOUR-APIFY-API-TOKEN" -os.environ["OPENAI_API_KEY"] = "YOUR-OPENAI-API-KEY" +The dataset components and `ApifyWebSearch` turn dataset items into documents with `default_dataset_mapping_function`. It takes the `markdown` or `text` field of website and search Actors as the content, and the page `url`, `title`, and `description` as `meta`. For other Actors, the content is the whole item as JSON. -document_loader = ApifyDatasetFromActorCall( - actor_id="apify/website-content-crawler", - run_input={ - "maxCrawlPages": 3, # limit the number of pages to crawl - "startUrls": [{"url": "https://haystack.deepset.ai/"}], - }, - dataset_mapping_function=lambda item: Document(content=item["text"] or "", meta={"url": item["url"]}), -) +To build the documents differently, pass your own mapping function: -document_store = InMemoryDocumentStore() -print(f"Initialized InMemoryDocumentStore with {document_store.count_documents()} documents") +```python +from haystack import Document -document_splitter = DocumentSplitter(split_by="word", split_length=150, split_overlap=50) -document_embedder = OpenAIDocumentEmbedder() -document_writer = DocumentWriter(document_store) +from apify_haystack import ApifyDatasetFromActorCall -pipe = Pipeline() -pipe.add_component("document_loader", document_loader) -pipe.add_component("document_splitter", document_splitter) -pipe.add_component("document_embedder", document_embedder) -pipe.add_component("document_writer", document_writer) -pipe.connect("document_loader", "document_splitter") -pipe.connect("document_splitter", "document_embedder") -pipe.connect("document_embedder", "document_writer") +def to_document(item: dict) -> Document: + return Document(content=item["text"], meta={"url": item["url"], "language": item["metadata"]["languageCode"]}) -print("\nCrawling will take some time ...") -print("You can visit https://console.apify.com/actors/runs to monitor the progress\n") -pipe.run({}) -print(f"Added {document_store.count_documents()} to vector from Website Content Crawler") +crawler = ApifyDatasetFromActorCall( + actor_id="apify/website-content-crawler", + dataset_mapping_function=to_document, + run_input={"startUrls": [{"url": "https://docs.haystack.deepset.ai/docs/intro"}], "maxCrawlPages": 10}, +) +``` -print("\n ### Retrieving documents from the document store using BM25 ###\n") -print("query='Haystack'\n") +## Run components asynchronously -bm25_retriever = InMemoryBM25Retriever(document_store) +Every component has `run_async`, and `ApifyActorTool` works with `Agent.run_async`. `Pipeline.run_async` uses them to wait for Actor runs without blocking the event loop. This example runs the `indexing` pipeline from [Answer questions about website content](#answer-questions-about-website-content): -for doc in bm25_retriever.run("Haystack", top_k=1)["documents"]: - print(doc.content) +```python +import asyncio -print("\n ### Retrieving documents from the document store using vector similarity ###\n") -retrieval_pipe = Pipeline() -retrieval_pipe.add_component("embedder", OpenAITextEmbedder()) -retrieval_pipe.add_component("retriever", InMemoryEmbeddingRetriever(document_store, top_k=1)) +result = asyncio.run(indexing.run_async({})) +``` -retrieval_pipe.connect("embedder.embedding", "retriever.query_embedding") +When an Actor or task run finishes with any status other than `SUCCEEDED`, the component raises a `RuntimeError` and loads no documents. -results = retrieval_pipe.run({"embedder": {"text": "What is Haystack?"}}) +## Save and load pipelines -for doc in results["retriever"]["documents"]: - print(doc.content) -``` +The components and the tool support [Haystack serialization](https://docs.haystack.deepset.ai/docs/serialization). They save the API token as the name of its environment variable, never as its value. Haystack only loads classes and functions from modules you trust, so allow `apify_haystack` and the module with your mapping function, if you use one. This example saves and restores the `indexing` pipeline from [Answer questions about website content](#answer-questions-about-website-content): -To run it, you can use the following command: `python apify_integration.py` +```python +from haystack import Pipeline - +yaml = indexing.dumps() +restored = Pipeline.loads(yaml, allowed_modules=["apify_haystack", "my_project.mapping"]) +``` ## Resources -- [Apify-haystack integration documentation](https://haystack.deepset.ai/integrations/apify) -- [Apify-haystack integration source code](https://github.com/apify/apify-haystack) -- [Example: RAG - Extract and use website content for question answering](https://haystack.deepset.ai/cookbook/apify_haystack_rag) -- [Example: RAG: Web Search and Analysis with Apify and Haystack](https://haystack.deepset.ai/cookbook/apify_haystack_rag_web_browser) -- [Example: Analyze Your Instagram Comments’ Vibe](https://haystack.deepset.ai/cookbook/apify_haystack_instagram_comments_analysis) +- [Apify integration on the Haystack website](https://haystack.deepset.ai/integrations/apify) +- [`apify-haystack` source code and examples](https://github.com/apify/apify-haystack) +- [Example: Extract and use website content for question answering](https://haystack.deepset.ai/cookbook/apify_haystack_rag) +- [Example: Web search and analysis with Apify and Haystack](https://haystack.deepset.ai/cookbook/apify_haystack_rag_web_browser) +- [Example: Analyze your Instagram comments' vibe](https://haystack.deepset.ai/cookbook/apify_haystack_instagram_comments_analysis)