hadoop - Optimised Hive query with JOIN , having million records -

-

मेरे पास 2 टेबल हैं - 

  bpm_agent_data - 40 मिलियन रिकॉर्ड, 5 कॉलम bpm_loan_data - 20 लाख रिकॉर्ड, 5 कॉलम

अब मैंने हाइव में एक प्रश्न चलाया - 

  चुनें count (bpm_agent_data.AgentID), count (bpm_loan_data.LoanNumber) bpm_agent_data से JOIN bpm_loan_data जहां bpm_loan_data.id = bpm_agent_data.id;

जो पूरा करने के लिए लंबा समय ले रहा है। HIVE में क्वेरी लिखने का आदर्श तरीका क्या होना चाहिए ताकि Reducer को इतना समय न लेना चाहिए।

उपरोक्त क्वेरी के लिए समाधान मिला, जहां पर 

  काउंट (bpm_agent_data.AgentID), गिनती (bpm_loan_data.LoanNumber) से bpm_agent_data पर जोड़ें bpm_loan_data पर (bpm_loan_data.id = bpm_agent_data.id );

Comments

Post a Comment

Popular posts from this blog

Pass DB Connection parameters to a Kettle a.k.a PDI table Input step dynamically from Excel -

-
Image
I have a requirement whenever I run my own kettle, dynamically took database connection parameters from an excel source Every run should go. I have an Excel column names: hostname, username, database, password. I want to pass these connection parameters to my table input stage dynamically whenever it runs the job. This is what I was trying to do. You can get it Connection parameters from the DB reading source (for example Excel or a CSV file in my example) < / Li> Store the parameters in the variable Using the variable in your connection's settings. Setting Variables Create another conversion for (you can not do it in the same conversion Uses): set variables element variable: The element that reads / writes your data creates a new connection and sets the connection parameter using $ {variable name_} Note that you will need to type in the field field in the field $ {password} Also keep in mind that this can be a security problem as the
Read more

multithreading - PhantomJS-Node in a for Loop -

-
I am trying to implement a screen-scraps application that opens a URL in which a parameter changes as it is for a loop: var data = [ '100', '101', '102', '103', '104']; (Var indexA = 0; indexA & lt; data.length; indexA ++) for {Phantom Generation ({'port': freeport}, function (pH) {ph.create page (function (page) {page.open ("http: //...++ data [index B] +" ... " , Status {console.log ("open site:" + data [index B], position); / * OLD * / indexB ++; page evaluation (function () {// page scrape and an object Return}, function {console.log (result.dataRequest); // here is the problem}); / * new and fixed * / indexB ++} first console Logs it: Open Site: 100 Success Opens Site: 101 Success B Ola Site: 102 Success Opening Site: 103 Success Opened Site: 104 Success Which is good. Data is verified in scrap HTML and from index to array Matches. So in page 101 I scrap '101 and retrieve' and pu
Read more

c++ - MATLAB .m file to .mex file using Matlab Compiler -

-
I know that it can speak as a very famous topic, but something that helps me with it Not found This scenario is: I have my functions function, no matter what's inside, it's a generic function that may depend on other toolboxes. Does. Has input and output; I want to generate the same myFun.mex function; : Lib myFun This way I get some new files: libmyfun.c libmyfun.dll Libmyfun.exp libmyfun.exports libmyfun .h libmyfun.lib Reading around and I think that Creating a myfun.cpp that is very famous is included: / * gateway function * / zero max function (ent NLHS, MXARRA * PLS [], int NRHS, CONST M Xeray * PHS []) Here is the Variable Deals * / / * code here * /} I have tried to do this and are called Regular of Max Generation: Max ('- v', 'myfun.cpp') But I get many errors, and I should say that my myfun.cpp copy / Paste Function is where I have a lot of doubts ... especially on input / output management. My q
Read more