Projects.
- Talk2Doc[Python,HTML,CSS,JavaScript]Talk2Data is a centralized platform designed to simplify the way college students interact with their academic and professional data. It combines multiple sources like Notion and Gmail to track job applications, and uses LLMs with RAG to allow querying custom notes.
README.md
Talk2Data - Centralized Ecosystem for College Students
Overview
Talk2Data is a centralized platform designed to simplify the way college students interact with their academic and professional data. By integrating multiple sources like Notion and Gmail, the platform organizes critical information for better accessibility and management. With the use of Retrieval-Augmented Generation (RAG) and personalized Large Language Models (LLMs), the application offers an intelligent and seamless way to handle class notes and job applications in a single, unified interface.
Key Features
- Academic Management: Retrieve, embed, and query notes stored on Notion for a personalized and efficient experience.
- Job Application Tracking: Automate the collection and organization of job-related information from Gmail into a structured Notion database.
- Intelligent Querying: Use VertexAI for embedding data and querying notes with high relevance, ensuring personalized and accurate outputs.
- Cloud-Hosted: The platform is designed for scalability and performance, leveraging serverless functions for data processing.
Problem Statement
Managing fragmented data from multiple sources is a persistent challenge for students. This system integrates:
- Class Notes: Pulls user-provided notes from Notion, converts them into embeddings, and generates contextually relevant responses.
- Job Applications: Scans Gmail inboxes for updates like deadlines and interview statuses, and organizes them into a Notion database.
By centralizing these workflows, Talk2Data enhances productivity and empowers students to focus on their goals.
Existing Solutions and Limitations
Job Application Tracking
- Simplify and similar tools track applications but:
- Miss applications submitted outside of the browser.
- Lack lifecycle tracking (e.g., assessment, interviews, offers) without manual updates.
Retrieval-Augmented Generation for Notes
- Reor is an open-source RAG-based note app but lacks integration with broader ecosystems like job tracking.
Talk2Data bridges these gaps by offering a holistic and integrated approach for students.
Architecture
The system is divided into two primary workflows:
1. Notes Flow
- Tracks changes in Notion pages where students store class notes.
- Retrieves new/updated data via Notion APIs every 5 minutes.
- Converts updated data into embeddings using VertexAI and stores them for efficient querying.
- Handles user queries by retrieving relevant notes and formatting outputs via text-embedding-005 LLM.
Optimization:
- Uses periodic polling over webhooks to batch-process changes for multiple users efficiently.
- Serverless functions scale dynamically to handle embedding and querying tasks.
2. Application Tracker
- Scans Gmail inbox for job-related updates.
- Organizes data into Notion databases, tracking deadlines, interviews, and other stages automatically.
Setup Requirements:
- Google Authentication: Users grant permission for Gmail access to track job applications.
- Notion Tokens: Users provide tokens and Page IDs for notes and job application tracking.
Key Technologies
- Retrieval-Augmented Generation (RAG) for embedding and querying user-specific data.
- VertexAI for generating embeddings.
- Notion API for tracking changes and retrieving user content.
- Google APIs for email access and job application tracking.
- Serverless Functions: Scalable cloud-based processing for retrieval, embedding, and querying.
How to Use
Prerequisites
- A Gmail account for job application tracking.
- A Notion account with a setup for class notes and job tracking.
- Access to Google Cloud Platform and VertexAI for embeddings.
Steps
- Clone this repository.
- Configure your environment with your Google API and Notion credentials.
- Deploy the serverless functions on a cloud platform of your choice.
- Set up periodic triggers for data retrieval and processing.
Future Enhancements
- Expand beyond Notion and Gmail to integrate with additional platforms like Slack and Outlook.
- Add support for more advanced lifecycle tracking in job applications.
- Provide customizable dashboards for users to visualize academic and job-related insights.
Contributors
- Sitanshu Kushwaha
- Sanchit Sahay
- Neel Gandhi
- Kevin Vaishnav
For questions or contributions, feel free to reach out to us at:
{sak9813, ss19723, njg9191, knv2014}@nyu.edu
Acknowledgments
We acknowledge the efforts of the NYU community and our mentors for guiding us through this project. Special thanks to Google Cloud and Notion for their APIs, which made this platform possible.
- mta-ridership[Jupyter Notebook]NYC Subway Traffic & Rideship Pattern Prediction.
- [Rust]POSIX tail which allows grep on live logs using --follow.
README.md
trunk
tail but filter when following.
Why?
I needed a small tool for work that would make it easy to filter
tail -foutputs live. Decided to put one together.Usage
Added feature for "sieving". Will follow for file changes
trunk -s <filter> /path/to/fileUsual tail commands are compatible
trunk -n <number of lines> /path/to/filetrunk -f /path/to/filetrunk -hUsage: trunk.exe [OPTIONS] <FILE> Arguments: <FILE> Path of the file to tail/follow Options: -f, --follow Follow a file for live changes -s, --sieve <SIEVE> Phrase to filter new lines with. Will automatically enable [-f --follow] [default: ] -n, --num-lines <NUM_LINES> Number of lines from the end to tail [default: 5] -h, --help Print help -V, --version Print versionBuilding
cargo build - [TypeScript,JavaScript,Go]Google App Engine replica for development workflows.
README.md
gilbert
An assistant to help with managing microservices compatible with Google App Engine configurations, in dev.
Motivation
Google App Engine, and in general serverless microservice management being on the rise, it's not always very easy to replicate the same environments in development.
Gilbert is designed to read
app.yaml&dispatch.yamlusing the App Engine configuration, to runnodejsservices, and dispatch the routes to these.How to Run
Installation
npm i gilbertRunning in Dev
gilbert run dev [--configFilePath; default=config.yaml] [--port; default=5050]Current Features
- Process
package.jsonscripts for nodejs runtimes - Combatible with dispatch and app configuration yamls based on GAE specifications. Just set the Dev Proxy to
localhost:[port]. - Isolated Logs and Processes for each service
To-Do
- Migrating to Containers from the currently running 'notainers'
- Migrating Process Management binaries to GoLang
- Providing shell for each service
- File-watching
- Local Databases and FileSystems
- Possible Production ready mode with containerd and nginx(?)
- Process
- [Rust]Playground for exploring packet captures using pcap syscalls
README.md
rustle
a very basic packet sniffer
TODO:
- Come up with a funner application for this -- maybe turning it into a lib to read pcapnp files.
- Deal with more Packet info
- Allowing filtering from CLI/Query
- hawkeye-2020[JavaScript,Go,HTML,Dockerfile,Shell,CSS]Bi-annual scavenger hunt for IECSE.
README.md
Hawkeye 2020
Development
- Install node, npm, go, docker, docker-compose
- Docker requires Linux / macOS / Windows Professional
$ cd hawkeye-2020 $ npm install --prefix client $ cp .env.example .env- Set environment variables in
.env - Generate encryption keys and add them to
.env
$ ./Taskfile.sh genkeysTaskfile commands
./Taskfile.sh # (default) run hawk in development mode ./Taskfile.sh start ./Taskfile.sh genkeys # Generate keys ./Taskfile.sh rediscli # Access redis CLI ./Taskfile.sh dbshell # Access database shellElixirs according to Index
- 0 Extra Hint
- 1 Region Multiplier
- 2 Hangman
- 3 Skip Question
