Skip to content

Instantly share code, notes, and snippets.

View SharathHebbar's full-sized avatar
💭
Redesign, Rebuild, Reclaim

Sharath S Hebbar SharathHebbar

💭
Redesign, Rebuild, Reclaim
61 followers · 170 following
View GitHub Profile
@SharathHebbar
SharathHebbar / gist:84e2f65097be0b9cfd3a8750761fcf04
Created November 9, 2024 08:58
from pyspark.sql import SparkSession
# Initialize Spark Session
spark = SparkSession.builder \
.appName("LocalSparkSQL") \
.config("spark.sql.shuffle.partitions", "4") \
.getOrCreate()
@SharathHebbar
SharathHebbar / New.py
Created October 16, 2024 13:58
# Read the CSV (with the first row as data)
df = spark.read.format("csv").option("header", "false").load("/path/to/csvfile")
# Extract the first row as the header
new_header = df.first()
# Create a new DataFrame without the first row
df_without_first_row = df.filter(df["_c0"] != new_header["_c0"])
@SharathHebbar
SharathHebbar / new.py
Last active September 30, 2024 15:00
import os
from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import AzureOpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.chains import RetrievalQA
from langchain.llms import AzureOpenAI
# Step 1: Load PDF Document
def load_pdf(pdf_path):
@SharathHebbar
SharathHebbar / lang.py
Created September 30, 2024 09:15
import os
from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import AzureOpenAIEmbeddings
from langchain.vectorstores import Chroma
# Step 1: Load PDF Document
def load_pdf(pdf_path):
loader = PyPDFLoader(pdf_path)
documents = loader.load()
@SharathHebbar
SharathHebbar / main.py
Created September 27, 2024 08:46
Main
from flask import Flask, render_template, request, redirect, url_for
from langchain import LLMChain
from langchain.llms import OpenAI
from langchain.prompts import PromptTemplate
import spacy
app = Flask(__name__)
# Load spaCy's English model for entity recognition
@SharathHebbar
SharathHebbar / Chat.html
Created September 27, 2024 07:03
Chat
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Chat, Summarize & Translate Bot</title>
<script src="https://cdn.tailwindcss.com"></script>
<style>
.entity {
@SharathHebbar
SharathHebbar / floating16.md
Created January 24, 2024 05:50

ValueError: Your setup doesn't support bf16/gpu. You need torch>=1.10, using Ampere GPU with cuda>=11.0

Change bf16 to fp16 for non Ampere GPUs

@SharathHebbar
SharathHebbar / trl.md
Last active January 24, 2024 05:44
TRL Issue resolved 
! pip show datasets
Name: datasets
Version: 2.16.1
Summary: HuggingFace community-driven open-source library of datasets
Home-page: https://github.com/huggingface/datasets
Author: HuggingFace Inc.
@SharathHebbar
SharathHebbar / 4bit.md
Last active January 24, 2024 05:45
Pushing 4bit quantized model 
!pip install git+https://github.com/huggingface/transformers.git -q -U # transformers version:  4.37.0
!pip install git+https://github.com/huggingface/accelerate.git -q -U # accelerate version:  0.27.0
!pip install bitsandbytes # bitsandbytes version:  0.42.0
!pip install git+https://github.com/huggingface/peft.git -q -U # peft version: 0.7.2
@SharathHebbar
SharathHebbar / table2.md
Created November 18, 2023 16:16
Table2
R-3 F-3 M-3
Segments Scores Descriptions
Best Customer