Blog
Notes and experiments in data science, people analytics and LLMs, from 2018 to today.
Generate Podcast From Any Webpage Using NotebookLlama
NotebookLlama, an open source version of Google NotebookLM deploys Meta’s Llama recipe for NotebookLM on the Llama family of models. Previously I used the NotebookLM to generate a podcast. I decided to build it from scratch using NotebookLlama workflow to generate a podcast from webpage content. Workflow:
Build Knowledge Graph RAG with LlamaIndex from PDF Documents
Previously I built a LLM chatbot with PDF documents, using the Retrieval Augmented Generation (RAG) technique. Traditional RAG leverages vector database and search retrieval methods, which measure the similarity or relatedness between different entities (such as words or phrases) based on their high-dimensional vector representations, typically the cosine similarity. For example, in a vector representation,…
Build LLM Chatbot With PDF Documents
Have you applied for loans, grants, or financial assistance programs? Have you dedicated significant time to deciphering legal or contractual documents to understand regulations and rules? Are you looking to streamline the process of reviewing extensive requirement documents? Your solution is here. I have developed an LLM chatbot, supported by RAG, to provide prompt responses…
What Skills Do You Need to Become a Data Engineer
People often ask me what skills needed to become a data engineer. Before answering that question, let’s take a look at what data engineer do. According to Coursera Data engineering is the practice designing and building systems for collecting, storing, and analyzing data at scale. Data engineering has become the backbone of many applications across industries, and…
Exploratory Spatial Data Analysis (ESDA) – Spatial Autocorrelation
In exploratory data analysis (EDA), we often calculate correlation coefficients and present the result in a heatmap. Correlation coefficient measures the statistical relationship between two variables. The correlation value represents how the change in one parameter would impact the other, e.g. quantity of purchase vs price. Correlation analysis is a very important concept in the…
Can You Tell Someone’s Gender Based on Tweets?
Predicting someone’s demographic attributes based on limited amount of information available is always a hot topic. It is common to use people’s name, ethnicity, location, and pictures for training models that can tell gender. Can you actually guess someone’s gender only based on what they share on Twitter? We will explore this using NLP techniques…
What Skills Do You Need to Become an HR Analyst
For work reasons, I have opportunities to interact with HR analysts everyday. I am always curious what skills one would need to become an HR analyst. Below is an HR Analyst job summary from SHRM.org. Other names for HR Analyst include People Analytics Analyst, Workforce Analytics Specialist, Data Analyst – People Analytics, etc. The Human Resource…
Trump And Trudeau Twitter Analysis During COVID-19 Crisis Part 2
In our last post, we extract @realDonalTrump and @JustinTrudeau tweets, clean up the texts, and generate word clouds. In this article, we will build a Latent Dirichlet Allocation (LDA) model to study the topics of the hundreds of tweets posted by the two world leaders. Topic Modeling Topic modeling is an unsupervised machine learning technique which…
Trump And Trudeau Twitter Analysis During COVID-19 Crisis Part 1
During the COVID-19 pandemic, people take their worries, concerns, frustration, and loves to social media to share with the rest of the world about their feelings and thoughts. Twitter has become one of official channels where world leaders communicate with their supporters and followers. To understand what keep them busy, we extract tweets of two…
COVID-19 (Coronavirus) Power BI Dashboard
On March 11, the WHO declared the Novel Coronavirus outbreak a pandemic, a new disease that has spread around the world. Many countries have seen reported cases of the virus. To help track and understand the daily spread of the virus, I built this Power BI dashboard. It provides an overview of the confirmed and…
To Know What People Twitter About #Coronavirus In One Minute
Year 2020 is not off to a good start. The ongoing Coronavirus outbreak that originated in Wuhan, China has infected thousands of people worldwide and killed hundreds. Numbers are still rising everyday. With all the quarantine controls and vaccine development, hope this global epidemic will be soon under control. When we are facing such a…
Who Are the Top HR Analytics Influencers on Twitter
Visualizing Twitter social network of HRanalytics Everyday people use social media such as Twitter to share thoughts and ideas. People with similar interests come together and interact on the online platform by re-sharing or replying posts they like. By studying how people interact on social networks, it will help us understand how information is distributed…
Auto Generated Insights of 2019 HR Tech Conference Twitter – Part 2 (Topic Modeling)
In our last post, we extract #HRTechConf tweets, clean up the texts, and generate a word cloud that highlights some of the buzzwords from the conference. But, what are the tweets talking about? Without reviewing each of the 7,000 tweets, how could we find out the popular topics? Let’s explore and see if tweet topics…
Auto Generated Insights of 2019 HR Tech Conference Twitter – Part 1 (Word Cloud)
HR Technology Conference and Expo, world’s leading and largest conference for HR and IT professionals, just took place in Las Vegas, from Oct 1 – 4, 2019. An incredibly amount of HR technology topics were covered at the conference. Unfortunately not everyone could be there, including myself. Is it possible to tell what the buzzwords…
Things Employees Like and Dislike About Their Companies
I work in people analytics and have been wondering all the time what make employees feel great or bad about their companies. Is it money? Workload? Opportunities to grow? Or team around them? I know the answer depends on the company, but is there anything in common for companies that employees like or dislike the…
Web App For Border Crossing Wait Time Forecast – Part 2
Keywords: Web App, Flask, AJAX, API, AWS, Virtual Environment Previously, I built the Flask web app that runs on my local machine for predicting border crossing wait time. This time I’ll show how it gets deployed on AWS and becomes a public available web app. Here is the link to web app http://35.164.32.109:5000/ There is…
Web App For Border Crossing Wait Time Forecast – Part 1
Keywords: Facebook Prophet, Web App, Flask, AJAX, API, AWS About a year ago I built a predictive model for predicting border crossing wait time. There were a lot of feature manipulation and parameter tweaking. Although results were encouraging, I always wanted to simplify the process and also make the model available for public use. After…
Credit Card Fraud Detection Using SMOTE Technique
Outlier detection is is an interesting application of machine learning. The goal is to identify those useful data records that can accurately profile abnormal behavior of the system. However, in real life examples, such special data like fraud and spam takes very small percentage of overall data population, which imposes challenges for developing machine learning…
Organizational network analysis – an experimental study
In every organization, people build and rely on informally-built networks seeking for information, advice, and collaborations. Often the invisible people networks are different from the formal organization hierarchy. Uncovering the informal but effective networks and understanding how information in the organization flows become crucial and enormously valuable to organization leaders. In this article, we will…
People Analytics – Attrition Predictions
According to the U.S. Bureau of Labor Statistics, 4.5 years is the average amount of time employees stay with their company today. It hurts an organization’s financials and morale , considering the amount of time they spend training. Can management learn from the past attrition and manage to reduce turnovers? Answer is yes. We will…
Simple Skill-based Job Recommendation Engine
What are the most demanded skills for data scientists? Python, R, SQL, and the list goes on and on. There are many surveys and reports that show some good statistics on popular data skills. In this post, I am going to gather first-hand information by scraping data science jobs from indeed.ca, analyze top skills required…
How many bikes to be shared in Vancouver NEXT WEEK – Part 2
This is Part 2 of building predictive models on Vancouver bike share. Part 1 is here. Python code can be found on my GitHub. Model Training Training dataset contains hourly bike rentals for each day from 01/01/2017 to 07/24/2018. Two decision tree models were trained: Random Forest (RF) and Gradient Boosted Trees (GBM). They are well known…
How many bikes to be shared in Vancouver NEXT WEEK – Part 1
Despite of worldwide debates on bike sharing benefits and challenges, Vancouver launched its own bike sharing program in summer 2016, Mobi sponsored by Shaw Go. First bike share appeared in Amsterdam in 1960’s, and then was introduced to other big European cities. It has got popularized by the Chinese in the last decade – 13…
How much time will it take to cross the border at Peace Arch?
Living in Vancouver, it is so convenient driving cross the border and have some fun on the other side. However, if you have headache of waiting in the long border-crossing lines and getting stuck for almost an hour, you are not alone. We all know the basic strategies on best/worst days/hours to cross, for example,…
Why so many wildfires in BC lately
British Columbia wildfires are burning out of control! There were a number of air quality warnings issued cross BC in summer 2018. Smoky air flew all the way to Alberta and even cross the border to US. It made me curious what’s going on with BC wildfires and what the situation was used to be.…
Build CNN for facial expression recognition with TensorFlow Eager on Google Colab
Key learning elements: » Run experiments in Google Colab and access files on Google Drive » Build and evaluate a model using Tensorflow Eager mode » Build a Convolutional Neural Network (CNN) to recognize 7 facial expressions For this exercise we are going to build a CNN for facial expression recognition on fer2013 dataset, available on…