Tag “crawling” — 131

Octoparse MCP Server

octoparse.com

Official Octoparse MCP server for template discovery, cloud tasks, and structured data export.

Exa Search

smithery.ai

Fast, intelligent web search and web crawling. Get fresh information about libraries, APIs, and SDKs.

timers studio

smithery.ai

gemini-extension.json n'est pas un mécanisme d'auto-discovery. Les Gemini Extensions sont publiées via Google AI Studio / Extensions Gallery, pas via crawl du .well-known/. Ton fichier est invisible pour Gemini. 3. ChatGPT ai-plugin.json correspond à la spec OpenAI Plugins, dépréciée en 2024. Remplacée par les GPTs (publiés via GPT Store) et les Actions (configurées dans un GPT custom). Ton fichier est mort techniquement. 4. llms.txt Proposition de standar

Crawlinx

smithery.ai

Crawlinx is a hosted Model Context Protocol server that lets your AI agent audit a website's technical SEO and act on the results. It crawls a site (50 pages by default, up to 200 via max_pages), runs 156 technical-SEO checks, and returns an LLM-ready Markdown report with a prioritized fix plan — small enough to keep in context on a real site. ☁️ Remote server over Streamable HTTP — no local install, no API key, no signup 🔎 156 checks — titles, meta, canonicals, orphan pages

Bright Data

smithery.ai

One (free) MCP for the Web. Easily search, crawl, navigate, and extract websites without getting blocked. Ideal for discovering and retrieving structured insights from any public source - effortlessly and ethically. New features are live - check out what’s new: https://brightdata.com/ai/mcp-server/launch-week

Crawlgraph MCP

smithery.ai

MCP server for the CrawlGraph backlink-intelligence API. Gives any MCP client - Claude Desktop, Claude Code, Cursor, Cline, Zed, Windsurf - backlink lookups and competitor gap analysis built on the public Common Crawl webgraph (4.4B edges, 120M domains).

Site Scorer

smithery.ai

Analyze websites to determine how effectively they are structured for automated crawling and data extraction. Receive actionable insights and ready-to-use prompts to fix structural issues and improve content visibility. Track recent reports to monitor improvements in site accessibility and performance.

Nimble MCP Server

smithery.ai

The Nimble MCP Server gives AI agents the ability to search, extract, map, crawl, and structure data from any website in real time. It exposes Nimble’s full web data platform as MCP tools that any compatible AI client can use.

serpdive

smithery.ai

**Real-time web search built for AI agents.** One tool, `serpdive_search`, returns the actual content of the best pages — extracted, cleaned, and sized for LLMs — not a list of links to crawl. ## Why SERPdive On a [public, replayable 1,000-question benchmark](https://github.com/edendalexis/serpdive-benchmark) against Tavily: - **Same speed** (median ~1.7 s server-side) - **20.2% fewer tokens** fed to your LLM per search - 🏆 **60.7% of decided quality duels** won Same ques

Oxylabs MCP

smithery.ai

Official Oxylabs MCP server - scrape any URL with JavaScript rendering and geo-targeting, parse Google and Amazon results, plus AI Studio tools (AI scraper, crawler, browser agent, search). Runs on Oxylabs infrastructure.

TensorFeed x402 Base Reader

tensorfeed.ai

Read-only Base mainnet reader. Verifies x402 payment settlements + AFTA federation status.

com.ainetcafe/netcafe-almanac

ainetcafe.com

Tables and ledgers checked by arithmetic, not by a model. 24 tools. MCP 2026-07-28 ready.

dev.cz-agents/adis

cz-agents.dev

Czech VAT-payer reliability (ADIS / nespolehlivý plátce DPH) + registered bank accounts by DIČ.

DigestSEO mcp-geo — AI Visibility MCP Server

digestseo.com

DigestSEO mcp-geo: track brand citations across five AI search engines.

agency.goji/goji

goji.agency

Answers on AEO, SEO, web and brand from GOJI's published material. Melbourne, Australia.

Bowmark

bowmark.ai

Do things on live websites: prices, availability, quotes, bookings, anything behind a form or login.

ai.buywhere/catalog-api

buywhere.ai

Agent-native product catalog: 300M+ products across US+SEA. Search, compare, deals.

ai.inxy/seo-audit

inxy.ai

Free SEO & AI-search (AEO) audits for any site or Shopify store, with emailed weekly score reports.

AI Manga & Anime Generator - Mangii MCP

mangii.ai

Create manga and anime art from text. 24 styles, multi-panel stories, BYOK.

MotiBlog

motiblog.ai

Agent-first SEO research and publishing: competitors, SERPs, internal links, and Search Console.

SAP MCP Server

mcp.sap.oobeprotocol.ai

Solana-native MCP gateway for SAP, DeFi tools, SNS identity, and x402 payments.

Food Logger MCP

foodlog.sig.ai

Food logging, nutrition summaries, and meal photo calorie and macro estimates.

Timothe's Growth Calendar

timothe.ai

Plan, schedule and generate SEO articles on a content calendar, from your agent.

ai.usestring/web-access

usestring.ai

The most accurate web access API. Stop getting blocked.

Vaaya

vaaya.ai

Pay-per-call agent superpowers: media/video gen, product demos, research, GTM, scraping, compute.

Vertical Marketplace

verticalmarketplace.ai

Agent-to-agent marketplace: AI agents list and buy data, services and compute. Signed receipts.

APICK AI

apick.app

LLM chat, text summarization and AI image generation

app.crawlio/crawlio-browser

crawlio.app

Bridge a live Chrome browser to your agent: 150 tools for capture, extraction, and observation.

ToolSnap MCP

toolsnap.app

Deterministic AI agent microtools, no accounts/API keys. fetch_extract: 98% token cut. 38 tools.

cloud.fetcher/fetcher

fetcher.cloud

The web capability layer for AI agents: render, extract, DNS, SSL, WHOIS & more via x402.

ActableSite AI Crawler Monitor

actablesite.com

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

AIOProductOS MCP

aioproductos.com

AIOProductOS: revenue, feedback, work and analytics on one customer record. 71 tools, 3 apps.

com.claritybriefs/mcp

claritybriefs.com

Free cross-lingual news briefings for AI agents across 89 languages. Read-only, hosted.

Conqueror

conquerorapp.com

Turns SEO insight into page changes: keyword research, SERP and rank data, rewrites you approve.

ContrastAPI

api.contrastcyber.com

55 tools, 7 Resources, Sigma rules, email SPF/DMARC, MITRE, CVE/KEV, risk_score. No key.

EnConvert

enconvert.com

Unified web ingestion for AI agents: perceive, discover, search, extract, ingest, watch, convert.