Skip to content

docs: update the Haystack integration page for apify-haystack 1.0 #3116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vdusek wants to merge 4 commits into
base: master
Choose a base branch
from
Open

docs: update the Haystack integration page for apify-haystack 1.0 #3116

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e7b8da0
docs: update the Haystack integration page for apify-haystack 1.0
vdusek Oct 26, 2026
7b5cdc2
docs: apply review suggestions to the Haystack integration page
vdusek Oct 29, 2026
a4a89e6
docs: note that task_input is required for ApifyDatasetFromTaskCall
vdusek Oct 29, 2026
4d5b12a
docs: describe running a Haystack task with its saved input
vdusek Oct 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
292 changes: 170 additions & 122 deletions sources/platform/integrations/ai/haystack.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,187 +1,235 @@
---
title: Haystack integration
sidebar_label: Haystack
description: Learn how to integrate Apify with Haystack to crawl websites, compute text embeddings, and build retrieval-augmented generation pipelines with ease.
description: Integrate Apify with Haystack to load web content into RAG pipelines, search the web, and give Haystack agents access to any Actor from Apify Store.
slug: /integrations/haystack
---

import ThirdPartyDisclaimer from '@site/sources/_partials/_third-party-integration.mdx';

[Haystack](https://haystack.deepset.ai/) is an open source framework for building production-ready LLM applications, agents, advanced retrieval-augmented generative pipelines, and state-of-the-art search systems that work intelligently over large document collections. For more information on Haystack, visit its [documentation](https://docs.haystack.deepset.ai/docs/intro).
[Haystack](https://haystack.deepset.ai/) is an open-source framework by deepset for building LLM applications, retrieval-augmented generation (RAG) pipelines, and AI agents. To learn more, see the [Haystack documentation](https://docs.haystack.deepset.ai/docs/intro).

In this example, we'll use the [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor, which can deeply crawl websites such as documentation sites, knowledge bases, or blogs, and extract text content from the web pages.
Then, we'll use the `OpenAIDocumentEmbedder` to compute text embeddings and the `InMemoryDocumentStore` to store documents in a temporary in-memory database.
The last step will be to retrieve the most similar documents.
The [`apify-haystack`](https://pypi.org/project/apify-haystack/) package connects Haystack to the [Apify platform](https://apify.com/). It provides these components:

This example uses the Apify-Haystack Python integration published on [PyPi](https://pypi.org/project/apify-haystack/).
Before we start with the integration, we need to install all dependencies:
- `ApifyDatasetFromActorCall` runs an Actor, waits for it to finish, and loads its default dataset as Haystack `Document` objects.
- `ApifyDatasetFromTaskCall` does the same for a saved [Actor task](/actors/running/tasks).
- `ApifyDatasetLoader` loads an existing [dataset](/storage/dataset) by its ID.
- `ApifyWebSearch` searches the web with the [RAG Web Browser](https://apify.com/apify/rag-web-browser) Actor and returns the content of the top result pages.
- `ApifyActorTool` lets a Haystack `Agent` run any Actor and work with its results.

```bash
pip install apify-haystack haystack-ai
```
<ThirdPartyDisclaimer />

Import all required packages:
## Prerequisites

```python
from haystack import Document, Pipeline
from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
from haystack.components.preprocessors import DocumentCleaner, DocumentSplitter
from haystack.components.retrievers import InMemoryBM25Retriever, InMemoryEmbeddingRetriever
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.utils.auth import Secret
- Python 3.11 or newer
- An [Apify account](https://console.apify.com/sign-up) and your API token from the [Integrations](https://console.apify.com/settings/integrations) page in Apify Console
- An [OpenAI API key](https://platform.openai.com/account/api-keys) for the examples on this page

from apify_haystack import ApifyDatasetFromActorCall
```
Haystack supports many other model providers too.

Find your [Apify API token](https://console.apify.com/settings/integrations) and [OpenAI API key](https://platform.openai.com/account/api-keys) and initialize these into environment variable:
## Installation

```python
import os
Install Haystack and the Apify integration:

os.environ["APIFY_API_TOKEN"] = "YOUR-APIFY-API-TOKEN"
os.environ["OPENAI_API_KEY"] = "YOUR-OPENAI-API-KEY"
```bash
pip install haystack-ai apify-haystack
```

First, you need to create a document loader that will crawl the haystack website using the Website Content Crawler:
Set your Apify API token and OpenAI API key as environment variables:

```python
document_loader = ApifyDatasetFromActorCall(
actor_id="apify/website-content-crawler",
run_input={
"maxCrawlPages": 3, # limit the number of pages to crawl
"startUrls": [{"url": "https://haystack.deepset.ai/"}],
},
dataset_mapping_function=lambda item: Document(content=item["text"] or "", meta={"url": item["url"]}),
)
```bash
export APIFY_TOKEN="YOUR-APIFY-API-TOKEN"
export OPENAI_API_KEY="YOUR-OPENAI-API-KEY"
```

You can learn more about input parameters on the [Website Content Crawler inputs page](https://apify.com/apify/website-content-crawler/input-schema).
The dataset mapping function is described in more detail in the [Retrieval augmented generation example](https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/apify_haystack_rag.ipynb).
The components read the token from `APIFY_TOKEN`, or from `APIFY_API_TOKEN` if `APIFY_TOKEN` isn't set. To read it from another variable, pass a Haystack `Secret`, for example `apify_token=Secret.from_env_var("MY_APIFY_TOKEN")`.

Next, you can utilize the [Haystack pipeline](https://docs.haystack.deepset.ai/docs/pipelines), which helps you connect several processing components together. I
n this example, we connect the document loader with the document splitter, document embedder, and document writer components.
## Answer questions about website content

```python
document_store = InMemoryDocumentStore()
This example crawls a part of the Haystack documentation with [Website Content Crawler](https://apify.com/apify/website-content-crawler), stores the pages in a document store, and answers questions about them.

1. Crawl the pages, split them into chunks, compute their embeddings, and store them:

```python
from haystack import Pipeline
from haystack.components.embedders import OpenAIDocumentEmbedder
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore

from apify_haystack import ApifyDatasetFromActorCall

document_store = InMemoryDocumentStore()

crawler = ApifyDatasetFromActorCall(
actor_id="apify/website-content-crawler",
run_input={
"startUrls": [{"url": "https://docs.haystack.deepset.ai/docs/intro"}],
"maxCrawlPages": 10,
},
)

indexing = Pipeline()
indexing.add_component("crawler", crawler)
indexing.add_component("splitter", DocumentSplitter(split_by="word", split_length=150, split_overlap=50))
indexing.add_component("embedder", OpenAIDocumentEmbedder())
indexing.add_component("writer", DocumentWriter(document_store))
indexing.connect("crawler", "splitter")
indexing.connect("splitter", "embedder")
indexing.connect("embedder", "writer")

indexing.run({})
print(f"Stored {document_store.count_documents()} documents")
```

The crawler turns each page into a `Document` with the page Markdown as its content and the page `url`, `title`, and `description` in `meta`. To change what it crawls, see the [Website Content Crawler input](https://apify.com/apify/website-content-crawler/input-schema). The crawl can take a few minutes. You can follow it on the [Runs](https://console.apify.com/actors/runs) page in Apify Console.

1. Retrieve the chunks closest to a question and let the LLM answer it:

```python
from haystack.components.builders import ChatPromptBuilder
from haystack.components.embedders import OpenAITextEmbedder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.retrievers import InMemoryEmbeddingRetriever
from haystack.dataclasses import ChatMessage

template = [
ChatMessage.from_user(
"""Answer the question based on the documents.

Documents:
{% for document in documents %}
{{ document.content }}
{% endfor %}

Question: {{ question }}"""
)
]

document_splitter = DocumentSplitter(split_by="word", split_length=150, split_overlap=50)
document_embedder = OpenAIDocumentEmbedder()
document_writer = DocumentWriter(document_store)
querying = Pipeline()
querying.add_component("embedder", OpenAITextEmbedder())
querying.add_component("retriever", InMemoryEmbeddingRetriever(document_store, top_k=3))
querying.add_component("prompt_builder", ChatPromptBuilder(template=template, required_variables="*"))
querying.add_component("llm", OpenAIChatGenerator())
querying.connect("embedder.embedding", "retriever.query_embedding")
querying.connect("retriever", "prompt_builder.documents")
querying.connect("prompt_builder", "llm")

pipe = Pipeline()
pipe.add_component("document_loader", document_loader)
pipe.add_component("document_splitter", document_splitter)
pipe.add_component("document_embedder", document_embedder)
pipe.add_component("document_writer", document_writer)
question = "What is a Haystack pipeline?"
result = querying.run({"embedder": {"text": question}, "prompt_builder": {"question": question}})
print(result["llm"]["replies"][0].text)
```

pipe.connect("document_loader", "document_splitter")
pipe.connect("document_splitter", "document_embedder")
pipe.connect("document_embedder", "document_writer")
## Search the web

`ApifyWebSearch` runs the RAG Web Browser Actor, which searches Google, scrapes the top results, and returns their content as Markdown. It has the same outputs as other Haystack web search components, so you can use it as a drop-in replacement in a pipeline:

```python
from apify_haystack import ApifyWebSearch

search = ApifyWebSearch(top_k=3)
result = search.run(query="What is new in Haystack 3?")

for document in result["documents"]:
print(document.meta["url"], document.meta.get("title"))
```

Run all the components in the pipeline:
The `documents` output holds the result pages, and `links` holds their URLs. To pass more input to the Actor, use `search_params`, for example `search_params={"scrapingTool": "browser-playwright"}` for pages that need a browser to render.

## Use Actors as agent tools

`ApifyActorTool` turns any Actor into a tool for a Haystack `Agent`. The tool takes its parameters from the Actor's input schema and its description from the Actor's description, so the LLM knows how to call it:

```python
pipe.run({})
from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

from apify_haystack import ApifyActorTool

browser = ApifyActorTool("apify/rag-web-browser", dataset_items_limit=3, max_total_charge_usd=0.5)
agent = Agent(chat_generator=OpenAIChatGenerator(), tools=[browser])

result = agent.run([ChatMessage.from_user("What is new in Haystack 3?")])
print(result["last_message"].text)
```

:::note Crawling may take some time
Note that:

The Actor call may take some time as it crawls the Haystack website.
- `dataset_items_limit` caps the number of dataset items the tool returns, which keeps its output within the model's context window.
- `max_total_charge_usd` caps the cost of each run the agent starts with a [pay-per-event](/actors/running/actors-in-store#pay-per-event) Actor.
- The tool wraps the items it returns in markers that tell the LLM to treat them as untrusted external content, which helps guard against prompt injection in scraped pages.

:::
## Load data from tasks and datasets

After running the pipeline code, you can print the results
To run a saved Actor task, use `ApifyDatasetFromTaskCall` with the task ID. The task runs with its saved input. To change some of its fields for a run, pass them in the optional `task_input`:

```python
print(f"Added {document_store.count_documents()} to vector from Website Content Crawler")
from apify_haystack import ApifyDatasetFromTaskCall

print("Retrieving documents from the document store using BM25")
print("query='Haystack'")
bm25_retriever = InMemoryBM25Retriever(document_store)
for doc in bm25_retriever.run("Haystack", top_k=1)["documents"]:
print(doc.content)
task = ApifyDatasetFromTaskCall(task_id="YOUR-TASK-ID", task_input={"maxCrawlPages": 5})
documents = task.run()["documents"]
```

If you want to test the whole example, you can simply create a new file, `apify_integration.py`, and copy the whole code into it.
To load a dataset that already exists, for example the result of an earlier run, use `ApifyDatasetLoader`:

```python
import os
from apify_haystack import ApifyDatasetLoader

from haystack import Document, Pipeline
from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.retrievers import InMemoryBM25Retriever, InMemoryEmbeddingRetriever
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
loader = ApifyDatasetLoader(dataset_id="YOUR-DATASET-ID")
documents = loader.run()["documents"]
```

from apify_haystack import ApifyDatasetFromActorCall
## Map dataset items to documents

os.environ["APIFY_API_TOKEN"] = "YOUR-APIFY-API-TOKEN"
os.environ["OPENAI_API_KEY"] = "YOUR-OPENAI-API-KEY"
The dataset components and `ApifyWebSearch` turn dataset items into documents with `default_dataset_mapping_function`. It takes the `markdown` or `text` field of website and search Actors as the content, and the page `url`, `title`, and `description` as `meta`. For other Actors, the content is the whole item as JSON.

document_loader = ApifyDatasetFromActorCall(
actor_id="apify/website-content-crawler",
run_input={
"maxCrawlPages": 3, # limit the number of pages to crawl
"startUrls": [{"url": "https://haystack.deepset.ai/"}],
},
dataset_mapping_function=lambda item: Document(content=item["text"] or "", meta={"url": item["url"]}),
)
To build the documents differently, pass your own mapping function:

document_store = InMemoryDocumentStore()
print(f"Initialized InMemoryDocumentStore with {document_store.count_documents()} documents")
```python
from haystack import Document

document_splitter = DocumentSplitter(split_by="word", split_length=150, split_overlap=50)
document_embedder = OpenAIDocumentEmbedder()
document_writer = DocumentWriter(document_store)
from apify_haystack import ApifyDatasetFromActorCall

pipe = Pipeline()
pipe.add_component("document_loader", document_loader)
pipe.add_component("document_splitter", document_splitter)
pipe.add_component("document_embedder", document_embedder)
pipe.add_component("document_writer", document_writer)

pipe.connect("document_loader", "document_splitter")
pipe.connect("document_splitter", "document_embedder")
pipe.connect("document_embedder", "document_writer")
def to_document(item: dict) -> Document:
return Document(content=item["text"], meta={"url": item["url"], "language": item["metadata"]["languageCode"]})

print("\nCrawling will take some time ...")
print("You can visit https://console.apify.com/actors/runs to monitor the progress\n")

pipe.run({})
print(f"Added {document_store.count_documents()} to vector from Website Content Crawler")
crawler = ApifyDatasetFromActorCall(
actor_id="apify/website-content-crawler",
dataset_mapping_function=to_document,
run_input={"startUrls": [{"url": "https://docs.haystack.deepset.ai/docs/intro"}], "maxCrawlPages": 10},
)
```

print("\n ### Retrieving documents from the document store using BM25 ###\n")
print("query='Haystack'\n")
## Run components asynchronously

bm25_retriever = InMemoryBM25Retriever(document_store)
Every component has `run_async`, and `ApifyActorTool` works with `Agent.run_async`. `Pipeline.run_async` uses them to wait for Actor runs without blocking the event loop. This example runs the `indexing` pipeline from [Answer questions about website content](#answer-questions-about-website-content):

for doc in bm25_retriever.run("Haystack", top_k=1)["documents"]:
print(doc.content)
```python
import asyncio

print("\n ### Retrieving documents from the document store using vector similarity ###\n")
retrieval_pipe = Pipeline()
retrieval_pipe.add_component("embedder", OpenAITextEmbedder())
retrieval_pipe.add_component("retriever", InMemoryEmbeddingRetriever(document_store, top_k=1))
result = asyncio.run(indexing.run_async({}))
```

retrieval_pipe.connect("embedder.embedding", "retriever.query_embedding")
When an Actor or task run finishes with any status other than `SUCCEEDED`, the component raises a `RuntimeError` and loads no documents.

results = retrieval_pipe.run({"embedder": {"text": "What is Haystack?"}})
## Save and load pipelines

for doc in results["retriever"]["documents"]:
print(doc.content)
```
The components and the tool support [Haystack serialization](https://docs.haystack.deepset.ai/docs/serialization). They save the API token as the name of its environment variable, never as its value. Haystack only loads classes and functions from modules you trust, so allow `apify_haystack` and the module with your mapping function, if you use one. This example saves and restores the `indexing` pipeline from [Answer questions about website content](#answer-questions-about-website-content):

To run it, you can use the following command: `python apify_integration.py`
```python
from haystack import Pipeline

<ThirdPartyDisclaimer />
yaml = indexing.dumps()
restored = Pipeline.loads(yaml, allowed_modules=["apify_haystack", "my_project.mapping"])
```

## Resources

- [Apify-haystack integration documentation](https://haystack.deepset.ai/integrations/apify)
- [Apify-haystack integration source code](https://github.com/apify/apify-haystack)
- [Example: RAG - Extract and use website content for question answering](https://haystack.deepset.ai/cookbook/apify_haystack_rag)
- [Example: RAG: Web Search and Analysis with Apify and Haystack](https://haystack.deepset.ai/cookbook/apify_haystack_rag_web_browser)
- [Example: Analyze Your Instagram Comments’ Vibe](https://haystack.deepset.ai/cookbook/apify_haystack_instagram_comments_analysis)
- [Apify integration on the Haystack website](https://haystack.deepset.ai/integrations/apify)
- [`apify-haystack` source code and examples](https://github.com/apify/apify-haystack)
- [Example: Extract and use website content for question answering](https://haystack.deepset.ai/cookbook/apify_haystack_rag)
- [Example: Web search and analysis with Apify and Haystack](https://haystack.deepset.ai/cookbook/apify_haystack_rag_web_browser)
- [Example: Analyze your Instagram comments' vibe](https://haystack.deepset.ai/cookbook/apify_haystack_instagram_comments_analysis)
Loading