DokuWiki

This is an old revision of the document!

Introduction

Product recommendations are usually implemented by using collaborative filtering technique which recommends items by calculating similar items to the given item as well as finding similar users to whom recommendations are being made. This is very useful algorithm and is core of many big companies like amazon,netflix etc. But the major concern with collaborative filtering is that it doesn’t handle cold start problem(eg: If user has no enrollments) well so to handle this case we need content based filtering which suggests items based on user profile.So ideally a hybrid recommendation system which implements both collaborative and content based filtering systems is required. Lightfm is the library which implements the hybrid recommendation systems.

LIGHTFM Advantages

Lightfm is a python library which implements number of popular recommendation algorithms for both implicit and explicit feedback, including efficient implementation of BPR and WARP ranking losses.It also makes it possible to incorporate both item and user metadata into the traditional matrix factorization algorithms. It has following advantages over other libraries:

It provides recommendations for both implicit and explicit data where as many other libraries support only explicit data.
It provides a facility to track metrics like model accuracy,precision which is very helpful to understand the recommendation accuracy.
It can make recommendation by considering the user and item features data.

Implementation

Lightfm is a popular recommendation algorithms for both implicit and explicit feedback data.It incorporates both item and user metadata into the traditional matrix factorization algorithms. It represents each user and item as the sum of the latent representations of their features, thus allowing recommendations to generalise to new items (via item features) and to new users (via user features).Lightfm implementation is very easy and has enough documentation in the below link

http://lyst.github.io/lightfm/docs/home.html.

Lightfm implementation can be divided into following steps

prepare interactions,user and item features matrices.
Train model with the available data.
predict results using the trained model.

1)Preparing data matrices:

dataset.fit(users=(x.user_id for x in user_rows), items=(x.course_id for x in item_rows), user_features=([x.city,x.job_role,x.org_id] for x in user_rows)).
(interactions,weights) = dataset.build_interactions(data=([x.user_id,x.course_id] for x in enrollment_rows))
user_features = dataset.build_user_features¹⁾

¹⁾

(user.user_id, [user.job_role,user.org_id,user.city]) for user in user_rows),False) 2)Train model:

model = LightFM(no_components=30, k=1, n=20, loss='warp', learning_schedule='adagrad')
model.fit_partial(interactions, user_features=user_features, epochs=30)

3)Predict results:

results = new_model.predict(dataset.mapping()[0][user_id], np.arange(num_items), user_features=user_features, num_threads=1)
results = np.argsort(-results)

===== Sample POC Input and output: ===== 1) User Recommendation

Input
- Enter User Id
- 4
Output
1. MV Apparatus
2. ABB in the Solar Inverter Space
3. Line Protection Basics
4. PC Switches and Fusegear
5. MV Switchgear

2) Item Recommendation

Input
- Enter Item Id
- 15023
Output
1. ABB in the Solar Inverter Space
2. EP Distributors channel overview
3. EP Distributors Channel final test
4. EP OEMs Channel final test
5. EP Panel Builders Channel final test

Sample code: from cassandra.cluster import Cluster

   from cassandra.auth import PlainTextAuthProvider
   from lightfm import LightFM
   from lightfm.evaluation import precision_at_k
   from lightfm.evaluation import auc_score
   from lightfm.data import Dataset
   import numpy as np
 
   auth_provider = PlainTextAuthProvider(username='cassandra', password='cassandra')
   cluster = Cluster(['107.170.83.67'], auth_provider=auth_provider)
   session = cluster.connect()
   session.set_keyspace('ie_transact_e1')
   dataset = Dataset()
   user_rows = session.execute("select user_id,job_role,city,org_id from e1_user")
   item_rows = session.execute("select course_id,title from e1_course")
   enrollment_rows = session.execute("select user_id,course_id from enrollment_e1")
   dataset.fit(users=(x.user_id for x in user_rows), items=(x.course_id for x in  item_rows),     user_features=([x.city,x.job_role,x.org_id] for x in user_rows))
   num_users, num_items = dataset.interactions_shape()
   (interactions,weights) = dataset.build_interactions(data=([x.user_id,x.course_id] for x in enrollment_rows))
    user_features = dataset.build_user_features(((user.user_id, [user.job_role,user.org_id,user.city]) for user in user_rows),False)
    model = LightFM(no_components=30, k=1, n=20, loss='warp', learning_schedule='adagrad')
    model.fit_partial(interactions, user_features=user_features, epochs=30)
    results = new_model.predict(dataset.mapping()[0][user_id], np.arange(num_items), user_features=user_features, num_threads=1)
    results = np.argsort(-results)
    def similar_items(internal_idx, item_feats_mtx, model, N=10):
          item_representations = model.get_item_representations()[1]
          scores = item_representations.dot(item_representations[internal_idx, :])
          item_norms = np.linalg.norm(item_representations, axis=1)
          scores /= item_norms
          best = np.argpartition(scores, -N)[-N:]
          return sorted(zip(best, scores[best] / item_norms[internal_idx]),key=lambda x: -x[1])
    sim_items = similar_items(dataset.mapping()[2][item_id],item_features,model)

DokuWiki

Site Tools

Table of Contents

Introduction

LIGHTFM Advantages

Implementation

Page Tools