docker file add

Dockerfile

@@ -0,0 +1,31 @@
+FROM python:3.10-slim
+
+RUN apt-get update && \
+    apt-get install -yq tzdata build-essential && \
+    apt-get clean
+
+RUN ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
+
+WORKDIR /app
+
+COPY core/requirements.txt requirements.txt
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY core .
+
+# download and unzip PocketBase
+ADD https://github.com/pocketbase/pocketbase/releases/download/v0.22.13/pocketbase_0.22.13_linux_amd64.zip /tmp/pb.zip
+# for arm user
+# ADD https://github.com/pocketbase/pocketbase/releases/download/v0.22.13/pocketbase_0.22.13_linux_arm64.zip /tmp/pb.zip
+RUN unzip /tmp/pb.zip -d /pb/
+
+RUN mkdir -p /pb
+
+COPY ./pb/pb_migrations /pb/pb_migrations
+COPY ./pb/pb_hooks /pb/pb_hooks
+COPY --from=builder /app/dist /pb/pb_public
+
+EXPOSE 8090
+EXPOSE 8077
+
+CMD tail -f /dev/null

README.md

@@ -95,6 +95,8 @@ WiseFlow has virtually no hardware requirements, with minimal system overhead, a
 
 SiliconFlow online inference service is compatible with the OpenAI SDK and provides open-source services for the above three models. Just configure LLM_API_BASE as "https://api.siliconflow.cn/v1" and set up LLM_API_KEY to use it.
 
+😄 Or you may prefer to use my [invitation link](https://cloud.siliconflow.cn?referrer=clx6wrtca00045766ahvexw92), so I can also get more token rewards 😄
+
 
 4. **Local Deployment**
 

README_CN.md

@@ -95,6 +95,8 @@ https://github.com/TeamWiseFlow/wiseflow/assets/96130569/bd4b2091-c02d-4457-9ec6
 ⚠️ 同时强烈推荐使用 **SiliconFlow** 的在线推理服务，更低的价格、更快的速度、更高的免费额度！⚠️ 
 
 SiliconFlow 在线推理服务兼容openai SDK，并同时提供上述三个模型的开源服务，仅需配置 LLM_API_BASE 为 "https://api.siliconflow.cn/v1" ， 并配置 LLM_API_KEY 即可使用。
+
+ 😄 或者您愿意使用我的[邀请链接](https://cloud.siliconflow.cn?referrer=clx6wrtca00045766ahvexw92)，这样我也可以获得更多token奖励  😄
     
     
 4. **本地部署**

README_DE.md

@@ -93,6 +93,8 @@ WiseFlow hat fast keine Hardwareanforderungen, minimale Systemlast und benötigt
 
 Der SiliconFlow Online-Inference-Service ist mit dem OpenAI SDK kompatibel und bietet Open-Service für die oben genannten drei Modelle. Konfiguriere LLM_API_BASE als "https://api.siliconflow.cn/v1" und LLM_API_KEY, um es zu verwenden.
 
+😄 Oder Sie möchten vielleicht meinen [Einladungslink](https://cloud.siliconflow.cn?referrer=clx6wrtca00045766ahvexw92) verwenden, damit ich auch mehr Token-Belohnungen erhalten kann 😄
+
 
 4. **Lokale Bereitstellung**
 

README_FR.md

@@ -96,6 +96,8 @@ WiseFlow n'a pratiquement aucune exigence matérielle, avec une empreinte systè
 
 Le service d'inférence en ligne SiliconFlow est compatible avec le SDK OpenAI et fournit des services open-source pour les trois modèles ci-dessus. Il suffit de configurer LLM_API_BASE comme "https://api.siliconflow.cn/v1" et de configurer LLM_API_KEY pour l'utiliser.
 
+😄 Ou peut-être préférez-vous utiliser mon [lien d'invitation](https://cloud.siliconflow.cn?referrer=clx6wrtca00045766ahvexw92), afin que je puisse également obtenir plus de récompenses en tokens 😄
+
 
 4. **Déploiement Local**
 

README_JP.md

@@ -93,6 +93,8 @@ https://github.com/TeamWiseFlow/wiseflow/assets/96130569/bd4b2091-c02d-4457-9ec6
 
 SiliconFlow のオンライン推論サービスはOpenAI SDKと互換性があり、上記の三つのモデルのオープンサービスも提供しています。LLM_API_BASE を "https://api.siliconflow.cn/v1" に設定し、LLM_API_KEY を設定するだけで使用できます。
 
+😄 もしよろしければ、私の[招待リンク](https://cloud.siliconflow.cn?referrer=clx6wrtca00045766ahvexw92)を使ってください。私もトークン報酬をさらに受け取ることができます 😄
+
 
 4. **ローカルデプロイメント**
 

compose.yaml

@@ -0,0 +1,16 @@
+services:
+  core:
+    build:
+      dockerfile: Dockerfile
+    image: WiseFlow/core
+    tty: true
+    stdin_open: true
+    entrypoint: bash docker_entrypoint.sh
+    env_file:
+      - .env
+    ports:
+      - 8090:8090
+      - 8077:8077
+    volumes:
+      - ./core/pb/pb_data:/pb/pb_data
+      - ./${PROJECT_DIR}:/app/${PROJECT_DIR}

core/scrapers/general_crawler.py

@@ -113,17 +113,19 @@ async def general_crawler(url: str, logger) -> tuple[int, Union[list, dict]]:
                     return -7, {}
 
         soup = BeautifulSoup(text, "html.parser")
+        # Note: The scheme used here is very crude, and it is recommended to write a separate parser for specific business scenarios
         # Parse all URLs
-        base_url = f"{parsed_url.scheme}://{domain}"
-        urls = set()
-        for link in soup.find_all("a", href=True):
-            absolute_url = urljoin(base_url, link["href"])
-            if urlparse(absolute_url).netloc == domain and absolute_url != url:
-                urls.add(absolute_url)
+        if len(url) < 50:
+            base_url = f"{parsed_url.scheme}://{domain}"
+            urls = set()
+            for link in soup.find_all("a", href=True):
+                absolute_url = urljoin(base_url, link["href"])
+                if urlparse(absolute_url).netloc == domain and absolute_url != url:
+                    urls.add(absolute_url)
 
-        if len(urls) > 21:
-            logger.info(f"{url} is more like an article list page, find {len(urls)} urls with the same netloc")
-            return 1, list(urls)
+            if len(urls) > 30:
+                logger.info(f"{url} is more like an article list page, find {len(urls)} urls with the same netloc")
+                return 1, list(urls)
 
     # 3. try to use gne to extract the information
     try: