(موتور جستجو با کتابخانه لوسین )Implement the baseline Lucene system for the News data

---

شرح پروژه

Implement the baseline Lucene system for the News data: 
This web site may help you: http://www.lucenetutorial.com/index.html
 
A. Preparation

Download the current release Lucene distribution from https://lucene.apache.org/core/downloads.html 
Test your distribution by following instructions in “Indexing Files” at https://lucene.apache.org/core/5_4_0/demo/overview-summary.html
Read and understand (enough to modify) the demo source code, mainly 

1. IndexFiles.java (https://lucene.apache.org/core/5_4_0/demo/src-html/org/apache/lucene/demo/IndexFiles.html): code to create a Lucene index.
2. SearchFiles.java (https://lucene.apache.org/core/5_4_0/demo/src-html/org/apache/lucene/demo/SearchFiles.html): code to search a Lucene index

Now you are ready to write your own indexer for the news data: 

1. You may start with the demo IndexFiles.java code
2. However: You will need to write your own document parser to process the files (to detect individual document boundaries within each file
3. Warning: Do not copy the input documents; Rather give the original location of the text directory as input to indexer and store the index in your own project directory

Make sure you index both the filename and the document id (look at one of the document in the collection to see the difference).
Pay attention to the tokenizer (analyzer in Lucene lingo) used. You will have to use the same one at search time.
Test your index with the modified SearchFiles.java program (hint: just make sure you can tell it where your index is, the rest can stay the same).
B.
B.1. Implement the TF*IDF ranking: 
Use Lucene ranking algorithm for finding top 10 documents
Implement TF-IDF ranking in Lucene and find top 10 documents.
Implement TF-IDF with different formulas for query and document.
 
B.2.  Variations of TF*IDF
 
1.     Think about two more variations of TF*IDF, try to implement them and test them
 
C. Evaluation
Trec_eval
1)     For the qrel provided in VU
a)      Create an appropriate format that is readable by trec_eval
2)     Run queries over dataset
3)     Output: for each topic, return up to 10 documents, one result per line, tab-delimited, in following format
       topic_id  \t Q0 \t document_id \t rank \t score \t your_login
e.g., 
       201  \t Q0 \t FBIS-41571 \t 143 \t 101.24 \t eugene      
4)     Use trec_eval to find measures such as MAP and P@5 and the pther measures
5)     Compare your algorithms

Documents: http://ir.dcs.gla.ac.uk/resources/test_collections/cran/
cran.all - The documents
cran.qry - The queries
cranqrel - The relevance assesments
readme - Some attempt at explanation especially about the relevance judgements


D. Report
1.     Prepare a report including at least 10 pages (single line, size 12) including the following parts:
a.      Introduction
                                                              i.      A summary of what you have done and your findings
b.      Indexing
                                                              i.      Create four indexes: with stop words, and without stop words, with stemming, without stemming. Explain the code, the difference in the size of indexes, the difference in the memory and the speed.

c.       Ranking Method
                                                              i.      An explanation of the implementation of the original TF-IDF, TF_IDF with different settings for query and documents, and your TWO new versions of TF-IDF
1.     You need to clearly justify your ideas, (you may used references to academic papers)
d.      Evaluation
                                                              i.      Tables and diagrams showing MAP, P@5, P@10 NDCG for all different ranking methods
                                                            ii.      Explain why your methods improve/worsen the original methods. Your data must be statistically significant (http://www.statisticssolutions.com/manova-analysis-paired-sample-t-test/, https://www.graphpad.com/quickcalcs/ttest1.cfm )
e.       Conclusion

این پروژه شامل 1 فایل مهم است، لطفا قبل از ارسال پیشنهاد حتما نسبت به بررسی این فایل اقدام فرمایید.

مهارت ها و تخصص های مورد نیاز

---

---