python - NLTK tokenize with collocations -

-

I am using NLTK and would like to summarize a text in relation to collocations: for example, to be "New York" Should have a single token, while the naïve token will split "New" and "York".

I know how and how to be found together, but how can not be aligned with both ...

Thank you.

approach, which seems right for you, called naming entity recognition, for nominated entity recognition There are many links dedicated to the NLTK. I just give you an example 

 nltk import sent_tokenize, word_tokenize, pos_tag, ne_chunk def extract_entities (text) from : entities = [] sent_tokenize for sentence (text): quantity = ne_chunk Returns entities __name__ == '__main__' if: text = "" "search for a multi agency (pos_tag (word_tokenize)) entities.extend ((part hasattr for part in quantity, if 'node']] Under the campaign, after the police, many states and the way of Mexico say that a former Los Angeles police officer There is a doubt in the killer of the college basketball coach and he is going through his oath to kill the police officers at the end of last week, killing one. "In this case, we are making our own goal," Sergeant. Rudy Lopez from the Corona Police Department said at a press conference. The suspect Christopher Jordan Dorner, 33 has been identified as, and he is considered extremely dangerous and equipped with many weapons, The murderers are believed to be revenge for the end of 2009 to give false statements from the Los Angeles Police Department, officials say that Dorner had published an online declaration in which "I am LAPD uniform In order to bring unconventional and uncomfortable war to those people, whether on duty or shut down. "" Print Extract_centity (text)

Output: 

  [Vr] Tree ('GPE', [('Mexico', 'NNP']], Tree ('GPE', [('Los', 'NNP'), ('Angels', 'NNP']], Tree ( 'Person', [('Rudy', 'NNP']), tree ('organization', [('loopage', 'nnp']], tree ('organization', [('corona', 'nnp '', 'Tree' ('person', [('Christopher', 'NNP'), ('Jordan', 'NNP'), ('Dorner', 'NNP']), tree ('GPE' Trees ('dosher', 'nnp'), tree ('gpi', [(' LAPD ',' NNP ')]]]

Another approach Testing - Using various measures of information overlap between the two random variables, such as mutual information, Pwaintvoari mutual information, t-test and others. & Lt; & Lt; >> Christopher D is a good start Manning and Heinrich Schuetz Chapter 5 collation is available for download. Here is an example of getting rid of the NLTK.


Comments

Post a Comment

Popular posts from this blog

Pass DB Connection parameters to a Kettle a.k.a PDI table Input step dynamically from Excel -

-
Image
I have a requirement whenever I run my own kettle, dynamically took database connection parameters from an excel source Every run should go. I have an Excel column names: hostname, username, database, password. I want to pass these connection parameters to my table input stage dynamically whenever it runs the job. This is what I was trying to do. You can get it Connection parameters from the DB reading source (for example Excel or a CSV file in my example) < / Li> Store the parameters in the variable Using the variable in your connection's settings. Setting Variables Create another conversion for (you can not do it in the same conversion Uses): set variables element variable: The element that reads / writes your data creates a new connection and sets the connection parameter using $ {variable name_} Note that you will need to type in the field field in the field $ {password} Also keep in mind that this can be a security problem as the
Read more

multithreading - PhantomJS-Node in a for Loop -

-
I am trying to implement a screen-scraps application that opens a URL in which a parameter changes as it is for a loop: var data = [ '100', '101', '102', '103', '104']; (Var indexA = 0; indexA & lt; data.length; indexA ++) for {Phantom Generation ({'port': freeport}, function (pH) {ph.create page (function (page) {page.open ("http: //...++ data [index B] +" ... " , Status {console.log ("open site:" + data [index B], position); / * OLD * / indexB ++; page evaluation (function () {// page scrape and an object Return}, function {console.log (result.dataRequest); // here is the problem}); / * new and fixed * / indexB ++} first console Logs it: Open Site: 100 Success Opens Site: 101 Success B Ola Site: 102 Success Opening Site: 103 Success Opened Site: 104 Success Which is good. Data is verified in scrap HTML and from index to array Matches. So in page 101 I scrap '101 and retrieve' and pu
Read more

c++ - MATLAB .m file to .mex file using Matlab Compiler -

-
I know that it can speak as a very famous topic, but something that helps me with it Not found This scenario is: I have my functions function, no matter what's inside, it's a generic function that may depend on other toolboxes. Does. Has input and output; I want to generate the same myFun.mex function; : Lib myFun This way I get some new files: libmyfun.c libmyfun.dll Libmyfun.exp libmyfun.exports libmyfun .h libmyfun.lib Reading around and I think that Creating a myfun.cpp that is very famous is included: / * gateway function * / zero max function (ent NLHS, MXARRA * PLS [], int NRHS, CONST M Xeray * PHS []) Here is the Variable Deals * / / * code here * /} I have tried to do this and are called Regular of Max Generation: Max ('- v', 'myfun.cpp') But I get many errors, and I should say that my myfun.cpp copy / Paste Function is where I have a lot of doubts ... especially on input / output management. My q
Read more