OpenCrawling is a high-performance enterprise data ingestion and security mapping framework. Powered by Java 25 and Spring Boot 4, it bridges enterprise repositories to vector search and LLM orchestrators while preserving document permissions (ACLs).
# Spin up PostgreSQL + pgvector, Redis, Ollama, Kafka
git clone https://github.com/OpenCrawling/opencrawling.git
cd opencrawling
docker compose up -d
Creating network "opencrawling_default" ... Created
Creating container opencrawling-postgres ... Running
Creating container opencrawling-redis ... Running
Creating container opencrawling-kafka ... Running
Creating container opencrawling-ollama ... Running
✔ All services started successfully on local ports.
Going far beyond simple scraping, OpenCrawling bridges data stores and vector indexes securely.
Built-in authority translation and Access Control List (ACL) mapping. Search engines only return documents that users are explicitly authorized to view.
Leverages Java 25 Virtual Threads and Spring Boot. Processes thousands of streams with minimal memory overhead, preventing source system bottlenecks.
Built-in semantic chunking, document cleanups, metadata enrichment, and vector embedding generation (using Ollama, OpenAI, or HuggingFace) on the fly.
Simple interfaces for writing custom connectors. Define a new Repository Source, Transformation filter, or Database Output connector in just a few lines of Java.
Decoupled, event-driven orchestration leveraging the Claim Check Pattern with Kafka and PostgreSQL.
Hover over or tap any node in the architecture diagram to view how OpenCrawling orchestrates the high-speed data flow and security controls.
Simulate real-time ingestion, security mapping, and embedding generation locally in your browser.
Select a source and click "Run Ingestion Job" to start simulation...
Follow these quick steps to get OpenCrawling up and running locally.
# Run from root of cloned repository to boot up all infrastructure containers
docker compose up -d
# Verify services are running
docker ps
# Services started:
# - PostgreSQL: localhost:5432 (pgvector ready)
# - Redis: localhost:6379 (session cache)
# - Apache Kafka: localhost:9092 (messaging topic)
# - Ollama: localhost:11434 (embedding engine)<!-- Add core connector API to your spring boot application -->
<dependency>
<groupId>org.opencrawling</groupId>
<artifactId>oc-core</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
<!-- Include FileSystem Repository Connector if required -->
<dependency>
<groupId>org.opencrawling</groupId>
<artifactId>oc-filesystem-repository-connector</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>// Implement RepositoryConnector interface to create custom sources
public class MyCustomRepositoryConnector implements RepositoryConnector {
@Override
public void connect(Map<String, String> configParams) throws RepositoryConnectionException {
// Establish authentication with target repository
}
@Override
public void scanDocuments(DocumentProcessor processor, JobContext context) {
// Scan documents incrementally and dispatch to processor
for (Document doc : fetchNewDocuments()) {
processor.processDocument(
doc.getId(),
doc.getContentStream(),
doc.getMetadata(),
doc.getAcl() // Synchronize Access Control Lists!
);
}
}
}We are an open-source organization building the future of secure AI ingestion pipelines.
Browse the OpenCrawling core engine source code, Java SPIs, connectors, and admin react UI.
View Repo →Explore all repositories, check out the organizational standards, templates, and profiles.
View Organization →Help shape the roadmap, report bugs, request new repository/output connectors, or ask questions.
Get Involved →