From e7b8da0459f9a856a0af301920719306747cfa96 Mon Sep 17 00:00:00 2001 From: Vlada Dusek Date: Mon, 26 Oct 2026 19:24:30 +0100 Subject: [PATCH 1/4] docs: update the Haystack integration page for apify-haystack 1.0 --- sources/platform/integrations/ai/haystack.md | 290 +++++++++++-------- 1 file changed, 168 insertions(+), 122 deletions(-) diff --git a/sources/platform/integrations/ai/haystack.md b/sources/platform/integrations/ai/haystack.md index f8b39da02c..1b3bd12a83 100644 --- a/sources/platform/integrations/ai/haystack.md +++ b/sources/platform/integrations/ai/haystack.md @@ -1,187 +1,233 @@ --- title: Haystack integration sidebar_label: Haystack -description: Learn how to integrate Apify with Haystack to crawl websites, compute text embeddings, and build retrieval-augmented generation pipelines with ease. +description: Integrate Apify with Haystack to load web content into RAG pipelines, search the web, and give Haystack agents access to any Actor from Apify Store. slug: /integrations/haystack --- import ThirdPartyDisclaimer from '@site/sources/_partials/_third-party-integration.mdx'; -[Haystack](https://haystack.deepset.ai/) is an open source framework for building production-ready LLM applications, agents, advanced retrieval-augmented generative pipelines, and state-of-the-art search systems that work intelligently over large document collections. For more information on Haystack, visit its [documentation](https://docs.haystack.deepset.ai/docs/intro). +[Haystack](https://haystack.deepset.ai/) is an open-source framework by deepset for building LLM applications, retrieval-augmented generation (RAG) pipelines, and AI agents. To learn more, see the [Haystack documentation](https://docs.haystack.deepset.ai/docs/intro). -In this example, we'll use the [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor, which can deeply crawl websites such as documentation sites, knowledge bases, or blogs, and extract text content from the web pages. -Then, we'll use the `OpenAIDocumentEmbedder` to compute text embeddings and the `InMemoryDocumentStore` to store documents in a temporary in-memory database. -The last step will be to retrieve the most similar documents. +The [`apify-haystack`](https://pypi.org/project/apify-haystack/) package connects Haystack to the [Apify platform](https://apify.com/). It provides these components: -This example uses the Apify-Haystack Python integration published on [PyPi](https://pypi.org/project/apify-haystack/). -Before we start with the integration, we need to install all dependencies: +- `ApifyDatasetFromActorCall` runs an Actor, waits for it to finish, and loads its default dataset as Haystack `Document` objects. +- `ApifyDatasetFromTaskCall` does the same for a saved [Actor task](/actors/running/tasks). +- `ApifyDatasetLoader` loads an existing [dataset](/storage/dataset) by its ID. +- `ApifyWebSearch` searches the web with the [RAG Web Browser](https://apify.com/apify/rag-web-browser) Actor and returns the content of the top result pages. +- `ApifyActorTool` lets a Haystack `Agent` run any Actor and work with its results. + + + +## Prerequisites + +- Python 3.11 or newer +- An [Apify account](https://console.apify.com/sign-up) and your API token from the [Integrations](https://console.apify.com/settings/integrations) page in Apify Console +- An [OpenAI API key](https://platform.openai.com/account/api-keys) for the examples on this page. Haystack supports many other model providers too. + +## Installation + +Install Haystack and the Apify integration: ```bash -pip install apify-haystack haystack-ai +pip install haystack-ai apify-haystack ``` -Import all required packages: +Set your Apify API token and OpenAI API key as environment variables: -```python -from haystack import Document, Pipeline -from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder -from haystack.components.preprocessors import DocumentCleaner, DocumentSplitter -from haystack.components.retrievers import InMemoryBM25Retriever, InMemoryEmbeddingRetriever -from haystack.components.writers import DocumentWriter -from haystack.document_stores.in_memory import InMemoryDocumentStore -from haystack.utils.auth import Secret - -from apify_haystack import ApifyDatasetFromActorCall +```bash +export APIFY_TOKEN="YOUR-APIFY-API-TOKEN" +export OPENAI_API_KEY="YOUR-OPENAI-API-KEY" ``` -Find your [Apify API token](https://console.apify.com/settings/integrations) and [OpenAI API key](https://platform.openai.com/account/api-keys) and initialize these into environment variable: +The components read the token from `APIFY_TOKEN`, or from `APIFY_API_TOKEN` if `APIFY_TOKEN` isn't set. To read it from another variable, pass a Haystack `Secret`, for example `apify_token=Secret.from_env_var("MY_APIFY_TOKEN")`. -```python -import os +## Answer questions about website content -os.environ["APIFY_API_TOKEN"] = "YOUR-APIFY-API-TOKEN" -os.environ["OPENAI_API_KEY"] = "YOUR-OPENAI-API-KEY" -``` +This example crawls a part of the Haystack documentation with [Website Content Crawler](https://apify.com/apify/website-content-crawler), stores the pages in a document store, and answers questions about them. -First, you need to create a document loader that will crawl the haystack website using the Website Content Crawler: +1. Crawl the pages, split them into chunks, compute their embeddings, and store them: -```python -document_loader = ApifyDatasetFromActorCall( - actor_id="apify/website-content-crawler", - run_input={ - "maxCrawlPages": 3, # limit the number of pages to crawl - "startUrls": [{"url": "https://haystack.deepset.ai/"}], - }, - dataset_mapping_function=lambda item: Document(content=item["text"] or "", meta={"url": item["url"]}), -) -``` + ```python + from haystack import Pipeline + from haystack.components.embedders import OpenAIDocumentEmbedder + from haystack.components.preprocessors import DocumentSplitter + from haystack.components.writers import DocumentWriter + from haystack.document_stores.in_memory import InMemoryDocumentStore -You can learn more about input parameters on the [Website Content Crawler inputs page](https://apify.com/apify/website-content-crawler/input-schema). -The dataset mapping function is described in more detail in the [Retrieval augmented generation example](https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/apify_haystack_rag.ipynb). + from apify_haystack import ApifyDatasetFromActorCall -Next, you can utilize the [Haystack pipeline](https://docs.haystack.deepset.ai/docs/pipelines), which helps you connect several processing components together. I -n this example, we connect the document loader with the document splitter, document embedder, and document writer components. + document_store = InMemoryDocumentStore() -```python -document_store = InMemoryDocumentStore() + crawler = ApifyDatasetFromActorCall( + actor_id="apify/website-content-crawler", + run_input={ + "startUrls": [{"url": "https://docs.haystack.deepset.ai/docs/intro"}], + "maxCrawlPages": 10, + }, + ) + + indexing = Pipeline() + indexing.add_component("crawler", crawler) + indexing.add_component("splitter", DocumentSplitter(split_by="word", split_length=150, split_overlap=50)) + indexing.add_component("embedder", OpenAIDocumentEmbedder()) + indexing.add_component("writer", DocumentWriter(document_store)) + indexing.connect("crawler", "splitter") + indexing.connect("splitter", "embedder") + indexing.connect("embedder", "writer") + + indexing.run({}) + print(f"Stored {document_store.count_documents()} documents") + ``` + + The crawler turns each page into a `Document` with the page Markdown as its content and the page `url`, `title`, and `description` in `meta`. To change what it crawls, see the [Website Content Crawler input](https://apify.com/apify/website-content-crawler/input-schema). The crawl can take a few minutes. You can follow it on the [Runs](https://console.apify.com/actors/runs) page in Apify Console. + +1. Retrieve the chunks closest to a question and let the LLM answer it: + + ```python + from haystack.components.builders import ChatPromptBuilder + from haystack.components.embedders import OpenAITextEmbedder + from haystack.components.generators.chat import OpenAIChatGenerator + from haystack.components.retrievers import InMemoryEmbeddingRetriever + from haystack.dataclasses import ChatMessage + + template = [ + ChatMessage.from_user( + """Answer the question based on the documents. + + Documents: + {% for document in documents %} + {{ document.content }} + {% endfor %} + + Question: {{ question }}""" + ) + ] -document_splitter = DocumentSplitter(split_by="word", split_length=150, split_overlap=50) -document_embedder = OpenAIDocumentEmbedder() -document_writer = DocumentWriter(document_store) + querying = Pipeline() + querying.add_component("embedder", OpenAITextEmbedder()) + querying.add_component("retriever", InMemoryEmbeddingRetriever(document_store, top_k=3)) + querying.add_component("prompt_builder", ChatPromptBuilder(template=template, required_variables="*")) + querying.add_component("llm", OpenAIChatGenerator()) + querying.connect("embedder.embedding", "retriever.query_embedding") + querying.connect("retriever", "prompt_builder.documents") + querying.connect("prompt_builder", "llm") -pipe = Pipeline() -pipe.add_component("document_loader", document_loader) -pipe.add_component("document_splitter", document_splitter) -pipe.add_component("document_embedder", document_embedder) -pipe.add_component("document_writer", document_writer) + question = "What is a Haystack pipeline?" + result = querying.run({"embedder": {"text": question}, "prompt_builder": {"question": question}}) + print(result["llm"]["replies"][0].text) + ``` -pipe.connect("document_loader", "document_splitter") -pipe.connect("document_splitter", "document_embedder") -pipe.connect("document_embedder", "document_writer") +## Search the web + +`ApifyWebSearch` runs the RAG Web Browser Actor, which searches Google, scrapes the top results, and returns their content as Markdown. It has the same outputs as other Haystack web search components, so you can use it as a drop-in replacement in a pipeline: + +```python +from apify_haystack import ApifyWebSearch + +search = ApifyWebSearch(top_k=3) +result = search.run(query="What is new in Haystack 3?") + +for document in result["documents"]: + print(document.meta["url"], document.meta.get("title")) ``` -Run all the components in the pipeline: +The `documents` output holds the result pages, and `links` holds their URLs. To pass more input to the Actor, use `search_params`, for example `search_params={"scrapingTool": "browser-playwright"}` for pages that need a browser to render. + +## Use Actors as agent tools + +`ApifyActorTool` turns any Actor into a tool for a Haystack `Agent`. The tool takes its parameters from the Actor's input schema and its description from the Actor's description, so the LLM knows how to call it: ```python -pipe.run({}) +from haystack.components.agents import Agent +from haystack.components.generators.chat import OpenAIChatGenerator +from haystack.dataclasses import ChatMessage + +from apify_haystack import ApifyActorTool + +browser = ApifyActorTool("apify/rag-web-browser", dataset_items_limit=3, max_total_charge_usd=0.5) +agent = Agent(chat_generator=OpenAIChatGenerator(), tools=[browser]) + +result = agent.run([ChatMessage.from_user("What is new in Haystack 3?")]) +print(result["last_message"].text) ``` -:::note Crawling may take some time +Note that: -The Actor call may take some time as it crawls the Haystack website. +- `dataset_items_limit` caps the number of dataset items the tool returns, which keeps its output within the model's context window. +- `max_total_charge_usd` caps the cost of each run the agent starts with a [pay-per-event](/actors/running/actors-in-store#pay-per-event) Actor. +- The tool wraps the items it returns in markers that tell the LLM to treat them as untrusted external content, which helps guard against prompt injection in scraped pages. -::: +## Load data from tasks and datasets -After running the pipeline code, you can print the results +To run a saved Actor task, use `ApifyDatasetFromTaskCall` with the task ID. The `task_input` overrides fields of the input saved in the task: ```python -print(f"Added {document_store.count_documents()} to vector from Website Content Crawler") +from apify_haystack import ApifyDatasetFromTaskCall -print("Retrieving documents from the document store using BM25") -print("query='Haystack'") -bm25_retriever = InMemoryBM25Retriever(document_store) -for doc in bm25_retriever.run("Haystack", top_k=1)["documents"]: - print(doc.content) +task = ApifyDatasetFromTaskCall(task_id="YOUR-TASK-ID", task_input={"maxCrawlPages": 5}) +documents = task.run()["documents"] ``` -If you want to test the whole example, you can simply create a new file, `apify_integration.py`, and copy the whole code into it. +To load a dataset that already exists, for example the result of an earlier run, use `ApifyDatasetLoader`: ```python -import os +from apify_haystack import ApifyDatasetLoader -from haystack import Document, Pipeline -from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder -from haystack.components.preprocessors import DocumentSplitter -from haystack.components.retrievers import InMemoryBM25Retriever, InMemoryEmbeddingRetriever -from haystack.components.writers import DocumentWriter -from haystack.document_stores.in_memory import InMemoryDocumentStore +loader = ApifyDatasetLoader(dataset_id="YOUR-DATASET-ID") +documents = loader.run()["documents"] +``` -from apify_haystack import ApifyDatasetFromActorCall +## Map dataset items to documents -os.environ["APIFY_API_TOKEN"] = "YOUR-APIFY-API-TOKEN" -os.environ["OPENAI_API_KEY"] = "YOUR-OPENAI-API-KEY" +The dataset components and `ApifyWebSearch` turn dataset items into documents with `default_dataset_mapping_function`. It takes the `markdown` or `text` field of website and search Actors as the content, and the page `url`, `title`, and `description` as `meta`. For other Actors, the content is the whole item as JSON. -document_loader = ApifyDatasetFromActorCall( - actor_id="apify/website-content-crawler", - run_input={ - "maxCrawlPages": 3, # limit the number of pages to crawl - "startUrls": [{"url": "https://haystack.deepset.ai/"}], - }, - dataset_mapping_function=lambda item: Document(content=item["text"] or "", meta={"url": item["url"]}), -) +To build the documents differently, pass your own mapping function: -document_store = InMemoryDocumentStore() -print(f"Initialized InMemoryDocumentStore with {document_store.count_documents()} documents") +```python +from haystack import Document -document_splitter = DocumentSplitter(split_by="word", split_length=150, split_overlap=50) -document_embedder = OpenAIDocumentEmbedder() -document_writer = DocumentWriter(document_store) +from apify_haystack import ApifyDatasetFromActorCall -pipe = Pipeline() -pipe.add_component("document_loader", document_loader) -pipe.add_component("document_splitter", document_splitter) -pipe.add_component("document_embedder", document_embedder) -pipe.add_component("document_writer", document_writer) -pipe.connect("document_loader", "document_splitter") -pipe.connect("document_splitter", "document_embedder") -pipe.connect("document_embedder", "document_writer") +def to_document(item: dict) -> Document: + return Document(content=item["text"], meta={"url": item["url"], "language": item["metadata"]["languageCode"]}) -print("\nCrawling will take some time ...") -print("You can visit https://console.apify.com/actors/runs to monitor the progress\n") -pipe.run({}) -print(f"Added {document_store.count_documents()} to vector from Website Content Crawler") +crawler = ApifyDatasetFromActorCall( + actor_id="apify/website-content-crawler", + dataset_mapping_function=to_document, + run_input={"startUrls": [{"url": "https://docs.haystack.deepset.ai/docs/intro"}], "maxCrawlPages": 10}, +) +``` -print("\n ### Retrieving documents from the document store using BM25 ###\n") -print("query='Haystack'\n") +## Run components asynchronously -bm25_retriever = InMemoryBM25Retriever(document_store) +Every component has `run_async`, and `ApifyActorTool` works with `Agent.run_async`. `Pipeline.run_async` uses them to wait for Actor runs without blocking the event loop: -for doc in bm25_retriever.run("Haystack", top_k=1)["documents"]: - print(doc.content) +```python +import asyncio -print("\n ### Retrieving documents from the document store using vector similarity ###\n") -retrieval_pipe = Pipeline() -retrieval_pipe.add_component("embedder", OpenAITextEmbedder()) -retrieval_pipe.add_component("retriever", InMemoryEmbeddingRetriever(document_store, top_k=1)) +result = asyncio.run(indexing.run_async({})) +``` -retrieval_pipe.connect("embedder.embedding", "retriever.query_embedding") +When an Actor or task run finishes with any status other than `SUCCEEDED`, the component raises a `RuntimeError` and loads no documents. -results = retrieval_pipe.run({"embedder": {"text": "What is Haystack?"}}) +## Save and load pipelines -for doc in results["retriever"]["documents"]: - print(doc.content) -``` +The components and the tool support [Haystack serialization](https://docs.haystack.deepset.ai/docs/serialization). They save the API token as the name of its environment variable, never as its value. Haystack only loads classes and functions from modules you trust, so allow `apify_haystack` and the module with your mapping function, if you use one: -To run it, you can use the following command: `python apify_integration.py` +```python +from haystack import Pipeline - +yaml = indexing.dumps() +restored = Pipeline.loads(yaml, allowed_modules=["apify_haystack", "my_project.mapping"]) +``` ## Resources -- [Apify-haystack integration documentation](https://haystack.deepset.ai/integrations/apify) -- [Apify-haystack integration source code](https://github.com/apify/apify-haystack) -- [Example: RAG - Extract and use website content for question answering](https://haystack.deepset.ai/cookbook/apify_haystack_rag) -- [Example: RAG: Web Search and Analysis with Apify and Haystack](https://haystack.deepset.ai/cookbook/apify_haystack_rag_web_browser) -- [Example: Analyze Your Instagram Comments’ Vibe](https://haystack.deepset.ai/cookbook/apify_haystack_instagram_comments_analysis) +- [Apify integration on the Haystack website](https://haystack.deepset.ai/integrations/apify) +- [`apify-haystack` source code and examples](https://github.com/apify/apify-haystack) +- [Example: Extract and use website content for question answering](https://haystack.deepset.ai/cookbook/apify_haystack_rag) +- [Example: Web search and analysis with Apify and Haystack](https://haystack.deepset.ai/cookbook/apify_haystack_rag_web_browser) +- [Example: Analyze your Instagram comments' vibe](https://haystack.deepset.ai/cookbook/apify_haystack_instagram_comments_analysis) From 7b5cdc2837c6e3521310b531c34f1abb757fd372 Mon Sep 17 00:00:00 2001 From: Vlada Dusek Date: Thu, 29 Oct 2026 13:07:09 +0100 Subject: [PATCH 2/4] docs: apply review suggestions to the Haystack integration page --- sources/platform/integrations/ai/haystack.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/sources/platform/integrations/ai/haystack.md b/sources/platform/integrations/ai/haystack.md index 1b3bd12a83..f7518adac7 100644 --- a/sources/platform/integrations/ai/haystack.md +++ b/sources/platform/integrations/ai/haystack.md @@ -23,7 +23,9 @@ The [`apify-haystack`](https://pypi.org/project/apify-haystack/) package connect - Python 3.11 or newer - An [Apify account](https://console.apify.com/sign-up) and your API token from the [Integrations](https://console.apify.com/settings/integrations) page in Apify Console -- An [OpenAI API key](https://platform.openai.com/account/api-keys) for the examples on this page. Haystack supports many other model providers too. +- An [OpenAI API key](https://platform.openai.com/account/api-keys) for the examples on this page + +Haystack supports many other model providers too. ## Installation @@ -203,7 +205,7 @@ crawler = ApifyDatasetFromActorCall( ## Run components asynchronously -Every component has `run_async`, and `ApifyActorTool` works with `Agent.run_async`. `Pipeline.run_async` uses them to wait for Actor runs without blocking the event loop: +Every component has `run_async`, and `ApifyActorTool` works with `Agent.run_async`. `Pipeline.run_async` uses them to wait for Actor runs without blocking the event loop. This example runs the `indexing` pipeline from [Answer questions about website content](#answer-questions-about-website-content): ```python import asyncio @@ -215,7 +217,7 @@ When an Actor or task run finishes with any status other than `SUCCEEDED`, the c ## Save and load pipelines -The components and the tool support [Haystack serialization](https://docs.haystack.deepset.ai/docs/serialization). They save the API token as the name of its environment variable, never as its value. Haystack only loads classes and functions from modules you trust, so allow `apify_haystack` and the module with your mapping function, if you use one: +The components and the tool support [Haystack serialization](https://docs.haystack.deepset.ai/docs/serialization). They save the API token as the name of its environment variable, never as its value. Haystack only loads classes and functions from modules you trust, so allow `apify_haystack` and the module with your mapping function, if you use one. This example saves and restores the `indexing` pipeline from [Answer questions about website content](#answer-questions-about-website-content): ```python from haystack import Pipeline From a4a89e6a7509049aa4950a905f3dfa59fda5c857 Mon Sep 17 00:00:00 2001 From: Vlada Dusek Date: Thu, 29 Oct 2026 13:07:17 +0100 Subject: [PATCH 3/4] docs: note that task_input is required for ApifyDatasetFromTaskCall --- sources/platform/integrations/ai/haystack.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sources/platform/integrations/ai/haystack.md b/sources/platform/integrations/ai/haystack.md index f7518adac7..9b8f25a22c 100644 --- a/sources/platform/integrations/ai/haystack.md +++ b/sources/platform/integrations/ai/haystack.md @@ -162,7 +162,7 @@ Note that: ## Load data from tasks and datasets -To run a saved Actor task, use `ApifyDatasetFromTaskCall` with the task ID. The `task_input` overrides fields of the input saved in the task: +To run a saved Actor task, use `ApifyDatasetFromTaskCall` with the task ID. The `task_input` overrides fields of the input saved in the task. It's required and can't be empty, so pass it to the constructor or to `run()`: ```python from apify_haystack import ApifyDatasetFromTaskCall From 4d5b12ad9ac374c8be796245dbedde76ff82e5b3 Mon Sep 17 00:00:00 2001 From: Vlada Dusek Date: Thu, 29 Oct 2026 13:38:34 +0100 Subject: [PATCH 4/4] docs: describe running a Haystack task with its saved input --- sources/platform/integrations/ai/haystack.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sources/platform/integrations/ai/haystack.md b/sources/platform/integrations/ai/haystack.md index 9b8f25a22c..4b653f137a 100644 --- a/sources/platform/integrations/ai/haystack.md +++ b/sources/platform/integrations/ai/haystack.md @@ -162,7 +162,7 @@ Note that: ## Load data from tasks and datasets -To run a saved Actor task, use `ApifyDatasetFromTaskCall` with the task ID. The `task_input` overrides fields of the input saved in the task. It's required and can't be empty, so pass it to the constructor or to `run()`: +To run a saved Actor task, use `ApifyDatasetFromTaskCall` with the task ID. The task runs with its saved input. To change some of its fields for a run, pass them in the optional `task_input`: ```python from apify_haystack import ApifyDatasetFromTaskCall