{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[],"gpuType":"T4"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"BTDII0dcwSkS"},"source":["# 🔧 Exhibition Connector RAG1 — Fixed & Colab-Ready\n","\n","این نسخه **اصلاح‌شده** کد RAG است که روی Google Colab اجرا می‌شود.\n","\n","**باگ‌های رفع شده:**\n","- مسیرهای import قدیمی langchain\n","- پارامتر `allow_dangerous_deserialization` در FAISS\n","- اسکیپ دوگانه `\\\\n` در پرامپت\n","- عدم مدیریت خطا\n","- مسیر نادرست فایل اکسل\n","- `get_relevant_documents` منسوخ شده → `invoke`"]},{"cell_type":"markdown","metadata":{"id":"-uRGaTqPwSkc"},"source":["## 📦 مرحله ۱: نصب کتابخانه‌ها"]},{"cell_type":"code","metadata":{"id":"8Q9aov5JwSke"},"source":["!pip install -q gradio langchain langchain-community langchain-text-splitters \\\n","    faiss-cpu beautifulsoup4 openpyxl transformers sentence-transformers \\\n","    accelerate torch aiohttp"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"pjd3RK-twSkh"},"source":["## 🔑 مرحله ۲: تنظیم توکن HuggingFace\n","\n","برای دسترسی به مدل Llama 3.1، توکن HuggingFace خود را وارد کنید.\n","\n","**روش توصیه شده:** روی آیکون 🔑 در نوار کناری Colab کلیک کنید و `HF_TOKEN` را اضافه کنید."]},{"cell_type":"code","metadata":{"id":"38kUiYiAwSki"},"source":["import os\n","\n","# روش ۱: از Colab Secrets (توصیه می‌شود)\n","try:\n","    from google.colab import userdata\n","    HF_TOKEN = userdata.get('HF_TOKEN')\n","    print(\"✅ توکن از Colab Secrets خوانده شد\")\n","except:\n","    # روش ۲: ورود دستی\n","    from getpass import getpass\n","    HF_TOKEN = getpass(\"🔑 توکن HuggingFace خود را وارد کنید: \")\n","    print(\"✅ توکن وارد شد\")\n","\n","os.environ[\"HF_TOKEN\"] = HF_TOKEN\n","assert HF_TOKEN, \"❌ توکن HuggingFace الزامی است!\"\n","print(\"✅ توکن تنظیم شد\")"],"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"FxyNP3a92suI"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"EG4dGiEL2soD"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"00ZdJ8v1wSkj"},"source":["## 📥 مرحله ۳: دانلود فایل اکسل"]},{"cell_type":"code","metadata":{"id":"adVNLDwQwSkl"},"source":["import requests\n","from pathlib import Path\n","\n","DATA_DIR = Path(\"/content/data\")\n","DATA_DIR.mkdir(exist_ok=True)\n","\n","xls_path = DATA_DIR / \"iran-oil_iran-oil_iran_oil.xlsx\"\n","xls_url = \"https://huggingface.co/spaces/sosa123454321/Exhibition-connector-rag1/resolve/main/iran-oil_iran-oil_iran%20oil.xlsx\"\n","\n","if not xls_path.exists():\n","    print(\"⏳ در حال دانلود فایل اکسل...\")\n","    r = requests.get(xls_url, timeout=60)\n","    r.raise_for_status()\n","    xls_path.write_bytes(r.content)\n","    print(f\"✅ فایل اکسل دانلود شد ({len(r.content):,} بایت)\")\n","else:\n","    print(f\"✅ فایل اکسل از قبل موجود است\")"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"j2Y_NjjxwSkm"},"source":["## 📄 مرحله ۴: بارگذاری اسناد"]},{"cell_type":"code","metadata":{"id":"FkiQuqvNwSkn"},"source":["from langchain_text_splitters import RecursiveCharacterTextSplitter\n","from langchain_community.document_loaders import WebBaseLoader\n","from langchain_core.documents import Document\n","import openpyxl\n","\n","print(\"⏳ در حال بارگذاری اسناد...\")\n","\n","# --- بارگذاری ویکی‌پدیا ---\n","wiki_url = 'https://fa.wikipedia.org/wiki/اوهیا'\n","wiki_loader = WebBaseLoader(wiki_url)\n","wiki_docs = wiki_loader.load()\n","print(f\"  ✅ ویکی‌پدیا: {len(wiki_docs)} سند\")\n","\n","# --- بارگذاری اکسل (با openpyxl مستقیم) ---\n","wb = openpyxl.load_workbook(str(xls_path))\n","excel_docs = []\n","for sheet_name in wb.sheetnames:\n","    ws = wb[sheet_name]\n","    rows = []\n","    for row in ws.iter_rows(values_only=True):\n","        rows.append(\"\\t\".join(str(c) if c is not None else \"\" for c in row))\n","    content = \"\\n\".join(rows)\n","    excel_docs.append(Document(\n","        page_content=content,\n","        metadata={\"source\": str(xls_path), \"sheet\": sheet_name}\n","    ))\n","    print(f\"  ✅ اکسل: شیت '{sheet_name}' — {ws.max_row} ردیف × {ws.max_column} ستون\")\n","\n","all_docs = wiki_docs + excel_docs\n","print(f\"\\n📄 مجموع اسناد: {len(all_docs)}\")\n","\n","# --- تقسیم به چانک‌ها ---\n","text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)\n","splits = text_splitter.split_documents(all_docs)\n","print(f\"✂️ تعداد چانک‌ها: {len(splits)}\")"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"UObO7u6hwSko"},"source":["## 🧠 مرحله ۵: ساخت ایندکس FAISS"]},{"cell_type":"code","metadata":{"id":"7HkoSkt3wSko"},"source":["from langchain_community.vectorstores import FAISS\n","from langchain_community.embeddings import HuggingFaceEmbeddings\n","import torch\n","\n","device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n","print(f\"⏳ در حال بارگذاری مدل Embedding روی {device}...\")\n","\n","embeddings = HuggingFaceEmbeddings(\n","    model_name=\"sentence-transformers/all-MiniLM-L6-v2\",\n","    model_kwargs={\"device\": device},\n","    encode_kwargs={\"normalize_embeddings\": True},\n",")\n","print(f\"✅ مدل Embedding بارگذاری شد\")\n","\n","faiss_index_path = DATA_DIR / \"faiss_index\"\n","\n","if faiss_index_path.exists():\n","    print(\"⏳ در حال بارگذاری ایندکس FAISS از حافظه...\")\n","    vectorstore = FAISS.load_local(\n","        str(faiss_index_path), embeddings, allow_dangerous_deserialization=True\n","    )\n","    print(f\"✅ ایندکس FAISS بارگذاری شد ({vectorstore.index.ntotal} بردار)\")\n","else:\n","    print(\"⏳ در حال ساخت ایندکس FAISS...\")\n","    vectorstore = FAISS.from_documents(splits, embeddings)\n","    vectorstore.save_local(str(faiss_index_path))\n","    print(f\"✅ ایندکس FAISS ساخته و ذخیره شد ({vectorstore.index.ntotal} بردار)\")\n","\n","retriever = vectorstore.as_retriever()\n","\n","# --- تست بازیابی ---\n","print(\"\\n🔍 تست بازیابی:\")\n","for q in [\"نمایشگاه نفت\", \"شرکت‌های پتروشیمی\"]:\n","    docs = retriever.invoke(q)\n","    print(f\"  جستجو: \\\"{q}\\\" → {len(docs)} نتیجه\")\n","    for i, d in enumerate(docs[:2]):\n","        print(f\"    [{i+1}] {d.page_content[:100].replace(chr(10), ' ')}...\")"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"LIkoSJgMwSkp"},"source":["## 🤖 مرحله ۶: بارگذاری مدل زبانی (LLM)\n","\n","مدل بر اساس GPU موجود انتخاب می‌شود:\n","- **GPU ≥ 16GB:** Llama 3.1 8B (بهترین کیفیت)\n","- **GPU ≥ 8GB:** TinyLlama 1.1B\n","- **بدون GPU:** GPT-2 (فقط برای تست)"]},{"cell_type":"code","metadata":{"id":"ntOPFmeiwSkp"},"source":["from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n","import torch\n","\n","# انتخاب مدل بر اساس GPU\n","if torch.cuda.is_available():\n","    gpu_mem = torch.cuda.get_device_properties(0).total_memory / (1024**3)\n","    print(f\"🖥️ GPU: {torch.cuda.get_device_name(0)} ({gpu_mem:.1f} GB)\")\n","\n","    if gpu_mem >= 16:\n","        LLM_MODEL = \"meta-llama/Meta-Llama-3.1-8B-Instruct\"\n","        USE_TOKEN = True\n","    elif gpu_mem >= 8:\n","        LLM_MODEL = \"TinyLlama/TinyLlama-1.1B-Chat-v1.0\"\n","        USE_TOKEN = False\n","    else:\n","        LLM_MODEL = \"gpt2\"\n","        USE_TOKEN = False\n","else:\n","    LLM_MODEL = \"gpt2\"\n","    USE_TOKEN = False\n","    print(\"⚠️ GPU در دسترس نیست! از مدل کوچک GPT-2 استفاده می‌شود\")\n","\n","print(f\"📦 مدل: {LLM_MODEL}\")\n","\n","print(f\"⏳ در حال بارگذاری مدل...\")\n","tokenizer = AutoTokenizer.from_pretrained(\n","    LLM_MODEL, token=HF_TOKEN if USE_TOKEN else None\n",")\n","llm_model = AutoModelForCausalLM.from_pretrained(\n","    LLM_MODEL,\n","    token=HF_TOKEN if USE_TOKEN else None,\n","    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,\n","    device_map=\"auto\" if torch.cuda.is_available() else None,\n",")\n","llm_pipe = pipeline(\n","    \"text-generation\",\n","    model=llm_model,\n","    tokenizer=tokenizer,\n","    max_new_tokens=512,\n","    max_length=llm_model.config.max_position_embeddings, # Explicitly set max_length to model's context window\n","    temperature=0.8,\n","    repetition_penalty=1.1,\n",")\n","print(f\"✅ مدل با موفقیت بارگذاری شد!\")"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"prVdrmsRwSkq"},"source":["## 🔗 مرحله ۷: تعریف پایپ‌لاین RAG"]},{"cell_type":"code","metadata":{"id":"8njAOLOFwSkq"},"source":["def llama_llm(question, context):\n","    \"\"\"تولید پاسخ با استفاده از مدل زبانی و متن بازیابی شده\"\"\"\n","    # System message to guide the LLM's behavior and output format\n","    system_message = (\n","        \"شما یک دستیار هوشمند نمایشگاه هستید که به سوالات کاربران در مورد شرکت‌ها، غرفه‌ها و اطلاعات نمایشگاه پاسخ می‌دهید. \"\n","        \"همیشه به زبان فارسی پاسخ دهید. \"\n","        \"پاسخ شما باید شامل اطلاعات دقیق از 'متن مرتبط' باشد. \"\n","        \"اگر شرکتی مرتبط با سوال پیدا کردید، اطلاعات آن را به صورت ساختاریافته ارائه دهید:\n","\"\n","        \"- **نام شرکت:** [نام شرکت]\\n\"\n","        \"- **فعالیت:** [خلاصه فعالیت]\\n\"\n","        \"- **سالن:** [شماره سالن]\\n\"\n","        \"- **غرفه:** [شماره غرفه]\\n\"\n","        \"- **وب‌سایت:** [آدرس کامل وب‌سایت، مثلاً https://example.com]\\n\"\n","        \"در صورتی که وب‌سایت در متن مرتبط وجود دارد، آن را به طور کامل در پاسخ خود قرار دهید تا `format_chatbot_output` بتواند آن را قابل کلیک کند. \"\n","        \"اگر اطلاعاتی یافت نشد، مودبانه اطلاع دهید.\"\n","    )\n","\n","    user_message = (\n","        f\"سوال: {question}\\n\\n\"\n","        f\"متن مرتبط:\\n{context}\\n\\n\"\n","        f\"لطفاً با توجه به دستورالعمل‌های بالا، پاسخ دقیق و کامل به زبان فارسی بدهید.\"\n","    )\n","\n","    prompt = f\"{system_message}\\n\\n{user_message}\"\n","\n","    # محدود کردن طول پرامپت برای جلوگیری از خطای overflow\n","    max_input_chars = 3000\n","    if len(prompt) > max_input_chars:\n","        # Truncate context if the prompt is too long, prioritizing system/user instructions\n","        # A more robust truncation might involve token count, but char count is simpler here.\n","        # This assumes context is the largest variable part.\n","        context_len = max_input_chars - (len(system_message) + len(user_message) - len(context))\n","        if context_len > 0:\n","            context = context[:context_len]\n","        prompt = f\"{system_message}\\n\\n{user_message.replace(f'متن مرتبط:\\n{context}', f'متن مرتبط:\\n{context}')}\"\n","        # Fallback if context_len is negative or small\n","        if len(prompt) > max_input_chars:\n","            prompt = prompt[:max_input_chars] + \"\\n\\nلطفاً پاسخ دهید:\"\n","\n","    output = llm_pipe(prompt)[0]['generated_text']\n","    return output[len(prompt):].strip()\n","\n","\n","def rag_chain(question):\n","    \"\"\"زنجیره RAG: بازیابی + تولید\"\"\"\n","    docs = retriever.invoke(question)\n","    context = \"\\n\\n\".join(doc.page_content for doc in docs)\n","    return llama_llm(question, context), docs\n","\n","\n","def get_important_facts(question):\n","    \"\"\"تابع اصلی پاسخ‌دهی\"\"\"\n","    if not question.strip():\n","        return \"لطفاً یک سوال معتبر وارد کنید.\"\n","    try:\n","        answer, docs = rag_chain(question)\n","        # Apply the new formatting function\n","        formatted_answer = format_chatbot_output(answer)\n","        return formatted_answer\n","    except Exception as e:\n","        return f\"خطایی رخ داد: {str(e)}\"\n","\n","print(\"✅ پایپ‌لاین RAG آماده است!\")"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"c68695b8"},"source":["import re\n","\n","def format_chatbot_output(text):\n","    \"\"\"Converts URLs in text to clickable markdown links.\"\"\"\n","    # Regex to find URLs (http, https, www. or just domain.tld)\n","    url_pattern = r\"\\b(?:https?://|www\\.)\\S+\\b|\\b[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,6}\\b(?:/[^\\s]*)?\"\n","\n","    def replace_url(match):\n","        url = match.group(0)\n","        if not url.startswith(('http://', 'https://')):\n","            if url.startswith('www.'):\n","                url = 'http://' + url\n","            else:\n","                # For cases like 'example.com', assume http\n","                url = 'http://' + url\n","        return f\"[{match.group(0)}]({url})\"\n","\n","    # Replace URLs with markdown links\n","    formatted_text = re.sub(url_pattern, replace_url, text)\n","\n","    return formatted_text\n","\n","print(\"✅ تابع فرمت‌دهی خروجی آماده شد!\")"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"d0d2120a"},"source":["Now, let's modify the `get_important_facts` function to use this new formatting. I'll modify cell `8njAOLOFwSkq`."]},{"cell_type":"markdown","metadata":{"id":"N0YTEwWFwSkq"},"source":["## 🧪 مرحله ۸: تست پایپ‌لاین RAG"]},{"cell_type":"code","metadata":{"id":"CH5WGDyNwSkr"},"source":["test_questions = [\n","    \"اطلاعات نفت ایران چیست؟\",\n","    \"شرکت‌های پتروشیمی کدامند؟\",\n","    \"نمایشگاه نفت تهران چه شرکت‌هایی دارد؟\",\n","]\n","\n","print(\"=\" * 60)\n","print(\"🧪 تست پایپ‌لاین RAG\")\n","print(\"=\" * 60)\n","\n","for q in test_questions:\n","    print(f\"\\n{'─'*50}\")\n","    print(f\"❓ سوال: {q}\")\n","\n","    try:\n","        docs = retriever.invoke(q)\n","        print(f\"📄 تعداد اسناد بازیابی شده: {len(docs)}\")\n","\n","        if docs:\n","            best = docs[0].page_content[:200].replace(\"\\n\", \" \")\n","            print(f\"🏆 بهترین نتیجه: {best}...\")\n","\n","        answer, _ = rag_chain(q)\n","        print(f\"🤖 پاسخ: {answer[:500]}\")\n","\n","    except Exception as e:\n","        print(f\"❌ خطا: {e}\")\n","\n","print(f\"\\n{'='*60}\")\n","print(\"✅ تست تمام شد!\")\n","print(\"=\" * 60)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"KgVvdLD5wSkr"},"source":["## 🖥️ مرحله ۹: رابط کاربری Gradio\n","\n","رابط کاربری وب اجرا می‌شود. لینک عمومی (`*.gradio.live`) ساخته می‌شود که می‌توانید به اشتراک بگذارید."]},{"cell_type":"code","metadata":{"id":"ScMX4awywSkr"},"source":["import gradio as gr\n","\n","WELCOME_MESSAGE = (\n","    \"سلام! من یک هوش مصنوعی هستم که برای کمک به شما در یافتن اطلاعات شرکت‌ها و \"\n","    \"نمایشگاه‌ها طراحی شده‌ام. هر سوالی درباره شرکت‌ها یا اطلاعات نمایشگاه دارید، بپرسید.\"\n",")\n","\n","iface = gr.Interface(\n","    fn=get_important_facts,\n","    inputs=gr.Textbox(lines=2, placeholder=\"سوال خود را اینجا وارد کنید...\"),\n","    outputs=\"text\",\n","    title=\"هوش مصنوعی پاسخگو به سوالات نمایشگاه\",\n","    description=WELCOME_MESSAGE,\n","    theme=\"default\",\n",")\n","\n","# share=True برای Colab تا لینک عمومی بسازد\n","iface.launch(share=True, server_name=\"0.0.0.0\", server_port=7860)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"5debafe6"},"source":["---"]},{"cell_type":"markdown","metadata":{"id":"548987af"},"source":["Now that the `format_chatbot_output` function is integrated, let's relaunch the Gradio interface to see the clickable links feature."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":715},"id":"0a2e8363","outputId":"509b31ab-f529-4137-b5e5-46503deedee8"},"source":["import gradio as gr\n","\n","WELCOME_MESSAGE = (\n","    \"سلام! من یک هوش مصنوعی هستم که برای کمک به شما در یافتن اطلاعات شرکت‌ها و \"\n","    \"نمایشگاه‌ها طراحی شده‌ام. هر سوالی درباره شرکت‌ها یا اطلاعات نمایشگاه دارید، بپرسید.\"\n",")\n","\n","iface_relaunch = gr.Interface(\n","    fn=get_important_facts,\n","    inputs=gr.Textbox(lines=2, placeholder=\"سوال خود را اینجا وارد کنید...\"),\n","    outputs=\"text\",\n","    title=\"هوش مصنوعی پاسخگو به سوالات نمایشگاه\",\n","    description=WELCOME_MESSAGE\n",")\n","\n","# share=True برای Colab تا لینک عمومی بسازد\n","iface_relaunch.launch(share=True, server_name=\"0.0.0.0\", debug=True, theme=\"default\")"],"execution_count":null,"outputs":[{"metadata":{"tags":null},"name":"stderr","output_type":"stream","text":["/usr/local/lib/python3.12/dist-packages/gradio/interface.py:171: UserWarning: The parameters have been moved from the Blocks constructor to the launch() method in Gradio 6.0: theme. Please pass these parameters to launch() instead.\n","  super().__init__(\n"]},{"metadata":{"tags":null},"name":"stdout","output_type":"stream","text":["Colab notebook detected. This cell will run indefinitely so that you can see errors and logs. To turn off, set debug=False in launch().\n","* Running on public URL: https://16f55a4e28b5c9189a.gradio.live\n","\n","This share link is temporary and will last for up to 1 week (best effort). For free permanent hosting and GPU upgrades, run `gradio deploy` from the terminal in the working directory to deploy to Hugging Face Spaces (https://huggingface.co/spaces)\n"]},{"data":{"text/html":["<div><iframe src=\"https://16f55a4e28b5c9189a.gradio.live\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"metadata":{"tags":null},"name":"stderr","output_type":"stream","text":["[transformers] Both `max_new_tokens` (=512) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)\n","[transformers] Both `max_new_tokens` (=512) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)\n"]},{"metadata":{"tags":null},"name":"stdout","output_type":"stream","text":["Created dataset file at: .gradio/flagged/dataset1.csv\n"]},{"metadata":{"tags":null},"name":"stderr","output_type":"stream","text":["[transformers] Both `max_new_tokens` (=512) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)\n"]}]}]}