Implement the baseline Lucene system for the News data

---

شرح پروژه

Implement the baseline Lucene system for the News data: 

This web site may help you: http://www.lucenetutorial.com/index.html

A. Preparation

1. Download the current release Lucene distribution from https://lucene.apache.org/core/downloads.html 
2. Test your distribution by following instructions in “Indexing Files” at https://lucene.apache.org/core/5_4_0/demo/overview-summary.html

1. Read and understand (enough to modify) the demo source code, mainly 

• IndexFiles.java (https://lucene.apache.org/core/5_4_0/demo/src-html/org/apache/lucene/demo/IndexFiles.html): code to create a Lucene index.
• SearchFiles.java (https://lucene.apache.org/core/5_4_0/demo/src-html/org/apache/lucene/demo/SearchFiles.html): code to search a Lucene index

1. Now you are ready to write your own indexer for the news data: 

• You may start with the demo IndexFiles.java code
• However: You will need to write your own document parser to process the files (to detect individual document boundaries within each file
• Warning: Do not copy the input documents; Rather give the original location of the text directory as input to indexer and store the index in your own project directory

Make sure you index both the filename and the document id (look at one of the document in the collection to see the difference). Pay attention to the tokenizer (analyzer in Lucene lingo) used. You will have to use the same one at search time. Test your index with the modified SearchFiles.java program (hint: just make sure you can tell it where your index is, the rest can stay the same).

B.

B.1. Implement the TF*IDF ranking: 

1. Use Lucene ranking algorithm for finding top 100 documents
2. Implement TF-IDF ranking in Lucene and find top 100 documents.
3. Implement TF-IDF with different formulas for query and document.

B.2.  Variations of TF*IDF

1. Think about two more variations of TF*IDF, try to implement them and test them

C. Evaluation

1. For the qrel provided in VU

   1. Create an appropriate format that is readable by trec_eval

2. Run queries over dataset

3. Output: for each topic, return up to 10 documents, one result per line, tab-delimited, in following format
          topic_id  \t Q0 \t document_id \t rank \t score \t your_login
   e.g., 
          201  \t Q0 \t FBIS-41571 \t 143 \t 101.24 \t eugene      

4. Use trec_eval to find measures such as MAP and P@5 and the pther measures

5. Compare your algorithms

   D. Report

1. Prepare a report including at least 10 pages (single line, size 12) including the following parts:

   1. Introduction

      1. A summary of what you have done and your findings

   2. Indexing

      1. Create four indexes: with stop words, and without stop words, with stemming, without stemming. Explain the code, the difference in the size of indexes, the difference in the memory and the speed.

1. 1. Ranking Method

      1. An explanation of the implementation of the original TF-IDF, TF_IDF with different settings for query and documents, and your TWO new versions of TF-IDF

         1. You need to clearly justify your ideas, (you may used references to academic papers)

   2. Evaluation

      1. Tables and diagrams showing MAP, P@5, P@10 for all different ranking methods

      2. Explain why your methods improve/worsen the original methods. Your data must be statistically significant (http://www.statisticssolutions.com/manova-analysis-paired-sample-t-test/, https://www.graphpad.com/quickcalcs/ttest1.cfm )

   3. Conclusion

این پروژه شامل 2 فایل مهم است، لطفا قبل از ارسال پیشنهاد حتما نسبت به بررسی این فایل اقدام فرمایید.

مهارت ها و تخصص های مورد نیاز

---

---