《textanalytics》课程简单总结(1):两种word relations——Paradigmatic vs. Syntagmatic
2015-07-18 10:37
190 查看
coursera上的公开课《https://www.coursera.org/course/textanalytics》系列,讲的非常不错哦。
1、两种关系:Paradigmatic vs. Syntagmatic(聚合和组合)
• Paradigmatic: A & B have paradigmatic relation if they can
be substituted for each other (i.e., A & B are in the same class)
– E.g., “cat” and “dog”; “Monday” and “Tuesday” (聚合:同一类别的,high similar context)
• Syntagmatic: A & B have syntagmatic relation if they can be combined with each other (i.e., A & B are related semantically)
– E.g., “cat” and “sit”; “car” and “drive”(组合:常在一起出现的,high correlated occurrences but relatively low individual occurrences)
2、挖掘Paradigmatic(聚合)关系:
2.1、如何挖掘两个词(例如dog和cat)的聚合关系强不强?
因为聚合关系本质上反映的是context
similarity,所以我们可以首先获取所有文档中出现dog、cat的句子的context,dog左边一个词的context、dog右边一个词的context,例如:Left1(“cat”)
= {“my”, “his”, “big”, “a”, “the”,…},Right1(“cat”) = {“eats”, “ate”, “is”, “has”, ….},Window(“cat”) = {“my”, “his”, “big”, “eats”, “fish”, …};同理可获得Left1(“dog”)
、Right1(“dog”)、Window(“dog”)
的context;这样,我们就可以通过计算Sim(“Cat”, “Dog”) = Sim(Left1(“cat”), Left1(“dog”)) + Sim(Right1(“cat”), Right1(“dog”)) + … + Sim(Window(“cat”), Window(“dog”))的大小来表示这两个词之间的聚合关系的强弱了。。。。
2.2具体到计算,常用的办法是Bag
of Words,也就是Vector Space Model (VSM),需要解决两个问题:
1)如何计算每一个向量,即把Left1(“cat”)
= {“my”, “his”, “big”, “a”, “the”,…}转化为vectorLeft1 = {3, 5, 8, 2, 7, ...}等VSM可用的形式。
2)如何计算Sim(x1,
x2)。
解决这两个问题的一般性办法:Expected
Overlap of Words in Context (EOWC):
d1=(x1,
…xN) ,其中xi =count(wi,d1)/|d1|
(从文档d1中随机选一个词,是wi的概率)
d2=(y1,
…yN) ,其中yi =count(wi,d2)/|d2| (从文档d2中随机选一个词,是wi的概率)
Sim(d1,d2)=d1.d2=
x1y1+...+xnyn(分别从d1、d2中随机选一个词,两个词一样的概率)
EOWC有两个主要问题:
– It
favors matching one frequent term very well over matching more distinct terms.
——通过平滑TF实现
情况1,d1、d2中的w1都非常频繁,其他wi却几乎不匹配,此时Sim(d1,d2)=10*10+0*0+...+1*3=123;情况2,d1、d2中的每个wi都不是非常频繁,但几乎都出现了几次,此时Sim(d1,d2)=5*5+4*3+...+2*6=111;对于这两种情况,EOWC是无法区分的,而我们更倾向于情况2代表的相似度!
– It treats every word equally (overlap on “the” isn’t as so meaningful as overlap on “eats”).
——通过IDF实现
通过平滑TF:BM25 Transformation
通过IDF:IDF Weighting
最终表达式:
-----
3、挖掘Syntagmatic(组合)关系:
参考下一篇博客:。
1、两种关系:Paradigmatic vs. Syntagmatic(聚合和组合)
• Paradigmatic: A & B have paradigmatic relation if they can
be substituted for each other (i.e., A & B are in the same class)
– E.g., “cat” and “dog”; “Monday” and “Tuesday” (聚合:同一类别的,high similar context)
• Syntagmatic: A & B have syntagmatic relation if they can be combined with each other (i.e., A & B are related semantically)
– E.g., “cat” and “sit”; “car” and “drive”(组合:常在一起出现的,high correlated occurrences but relatively low individual occurrences)
2、挖掘Paradigmatic(聚合)关系:
2.1、如何挖掘两个词(例如dog和cat)的聚合关系强不强?
因为聚合关系本质上反映的是context
similarity,所以我们可以首先获取所有文档中出现dog、cat的句子的context,dog左边一个词的context、dog右边一个词的context,例如:Left1(“cat”)
= {“my”, “his”, “big”, “a”, “the”,…},Right1(“cat”) = {“eats”, “ate”, “is”, “has”, ….},Window(“cat”) = {“my”, “his”, “big”, “eats”, “fish”, …};同理可获得Left1(“dog”)
、Right1(“dog”)、Window(“dog”)
的context;这样,我们就可以通过计算Sim(“Cat”, “Dog”) = Sim(Left1(“cat”), Left1(“dog”)) + Sim(Right1(“cat”), Right1(“dog”)) + … + Sim(Window(“cat”), Window(“dog”))的大小来表示这两个词之间的聚合关系的强弱了。。。。
2.2具体到计算,常用的办法是Bag
of Words,也就是Vector Space Model (VSM),需要解决两个问题:
1)如何计算每一个向量,即把Left1(“cat”)
= {“my”, “his”, “big”, “a”, “the”,…}转化为vectorLeft1 = {3, 5, 8, 2, 7, ...}等VSM可用的形式。
2)如何计算Sim(x1,
x2)。
解决这两个问题的一般性办法:Expected
Overlap of Words in Context (EOWC):
d1=(x1,
…xN) ,其中xi =count(wi,d1)/|d1|
(从文档d1中随机选一个词,是wi的概率)
d2=(y1,
…yN) ,其中yi =count(wi,d2)/|d2| (从文档d2中随机选一个词,是wi的概率)
Sim(d1,d2)=d1.d2=
x1y1+...+xnyn(分别从d1、d2中随机选一个词,两个词一样的概率)
EOWC有两个主要问题:
– It
favors matching one frequent term very well over matching more distinct terms.
——通过平滑TF实现
情况1,d1、d2中的w1都非常频繁,其他wi却几乎不匹配,此时Sim(d1,d2)=10*10+0*0+...+1*3=123;情况2,d1、d2中的每个wi都不是非常频繁,但几乎都出现了几次,此时Sim(d1,d2)=5*5+4*3+...+2*6=111;对于这两种情况,EOWC是无法区分的,而我们更倾向于情况2代表的相似度!
– It treats every word equally (overlap on “the” isn’t as so meaningful as overlap on “eats”).
——通过IDF实现
通过平滑TF:BM25 Transformation
通过IDF:IDF Weighting
最终表达式:
-----
3、挖掘Syntagmatic(组合)关系:
参考下一篇博客:。
相关文章推荐
- 类的扩展
- 自定义ViewGroup(这里以自定义一个类似ViewPager的控件为例)
- [译]用R语言做挖掘数据《三》
- web.py安装时出现ImportError: No module named 'utils'
- 《RocketMQ 安装和使用》
- SSL/TLS协议运行机制的概述
- 设计一个算法,采用BFS方式输出图G中从顶点u到v的最短路径(不带权的无向连通图G采用邻接表存储)
- LeetCode 118 Pascal's Triangle
- [LeetCode][Java] Minimum Window Substring
- 【实训项目】银行储蓄系统
- Secure Boot
- 地道
- SSL/TLS协议运行机制的概述
- linux运维人员需要知道的重要/常用目录介绍
- ajax提交form表单的两种方法
- 1048. Find Coins (25)
- [UEFI启动教程][第三章]BIOS锁定纯UEFI启动的解锁办法
- 3Sum
- wordpress添加后台菜单及权限设置
- java初学笔记20150718.十进制数转换为二进